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EXECUTIVE SUMMARY 



The District of Columbia School Choice Incentive Act of 2003 was passed by Congress in January 2004. 
The Act provided funds for District of Columbia Public Schools (DCPS) improvement activities and 
charter school facility acquisitions. Most notably, the statute established what is now called the DC 
Opportunity Scholarship Program — the first federal government initiative to provide K-12 education 
scholarships, or vouchers, to families to send their children to private schools of choice. 

The DC Opportunity Scholarship Program has the following programmatic elements: 

• To be eligible, students entering grades K-12 must reside in the District and have a family income 
at or below 185 percent of the federal poverty line. 

• Participating students will receive scholarships of up to $7,500 to cover the costs of tuition, 
school fees, and transportation to a participating private school of choice. 

• Scholarships are renewable for up to 5 years (as funds are appropriated), as long as students 
remain eligible for the program and remain in good academic standing at the private schools they 
are attending. 

• If there are more eligible applicants than available scholarships or open slots in private schools, 
applicants are to be awarded scholarships and admission to private schools random selection, for 
example by lottery. 

• Private schools participating in the program must be located in the District, and agree to program 
requirements regarding nondiscrimination in admissions, fiscal accountability, and cooperation 
with the evaluation. 

The Act requires that this 5-year scholarship program be rigorously evaluated by an independent research 
team, using the “strongest possible research design for determining the effectiveness” of the program and 
addressing a specific set of student comparisons and topics (Section 309). The evaluation thus has several 
components: (1) an impact analysis, comparing outcomes of eligible applicants (students and their 
parents) from public schools randomly assigned to receive or not receive a scholarship through a lottery, 
and (2) a performance reporting analysis, comparing all students participating in the scholarship program 
to students in the same grades in DCPS. All participating students includes those randomly assigned 
scholarships and those who received scholarships automatically, those who were attending public schools 
and those attending private schools when they entered the scholarship program. Because DCPS students 
who did not apply to the scholarship program are likely to be quite different from those who applied and 
are participating, the impact analysis will be the source of the reliable, causal evidence on program 
effectiveness called for in the legislation. 

This document is the first of a series of annual reports from the evaluation team, as mandated by 
Congress. Because the initial cohort of program participants — those who applied in spring 2004 to receive 
scholarships for the 2004-05 school year — just recently matriculated at their new schools, no impact 
information is available at this time. Instead, the report describes the purposes and design of the 
scholarship program, the first-year implementation activities that generated 1,848 eligible applicants and 
58 participating private schools, the process of awarding scholarships to 1,366 student applicants, and the 
characteristics of both applicants and scholarship users. The report provides an important foundation for 
the later examination of program impacts. 
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Program Implementation in 2004: Recruitment and Applications (Chapter 2) 

The Washington Scholarship Fund (WSF) was awarded a grant by the U.S. Department of Education 
(ED) Office of Innovation and Improvement, in partnership with the DC Mayor’s Office, to implement 
the program, starting in March 2004. Despite the challenges stemming from the late start of the program, 
the implemented recruited 58 schools to participate in the program in some capacity in 2004-05 and 
obtained applications from 1,848 students deemed eligible for the program. 



Participating Schools 

The 58 private schools participating in the program during its inaugural year represent 53 percent of all 
private schools in the District (Table ES-1). All but four of the schools made new slots in their schools 
available to scholarship winners. Four schools were willing to enroll scholarship students only if they had 
been accepted to the school for the 2004-05 school year prior to the launch of the scholarship program. 

Table ES-1. Number of DC Private Schools Participating in the DC Opportunity 
Scholarship Program: 2004-05 



Private Schools 


Number of 
Schools 


Percent 


In the District of Columbia 


109 


100 


Participating in the program in some capacity 


58 


53 


Set aside slots for DC Opportunity Scholarship Program 
participants 


54 


50 


Agreed to accept scholarships only for eligible students 
already admitted 


4 


4 



NOTE: Detail may not sum to totals because of rounding. 

SOURCE: “School Directory, D.C. K-12 Scholarship Program, 2004-05 School Year,” Washington 
Scholarship Fund, June 2004. 



The characteristics of the private schools that chose to participate in the program the first year include the 
following: 

• Mostly religiously affiliated: about 51 percent are affiliated with the Roman Catholic Church, 
another 21 percent are affiliated with various non-Catholic religions, and approximately 28 
percent are independent private schools. 

• Long established in the area: more than three-quarters of the schools have been in existence since 
1983 and the most recently established participating private school opened in 2002. 

• Already serve a high proportion of students of color: on average, 82 percent of students in 
participating private schools are from minority raciaFethnic groups, compared to 95 percent for 
DCPS schools. 

• On average, have lower school size and student-teacher ratios than do DCPS schools. 
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• Mostly (about 70 percent) charge tuitions that are under the $7,500 maximum provided through 
the federal scholarship program. 

• More likely to be religiously affiliated and serve higher proportions of students of color than are 
private schools that chose not to participate in the program the first year. Nonparticipating 
schools also tend to charge higher tuitions and have smaller class sizes than do private schools 
that are currently involved in the program. 



Participating Families and Students 

The program implementer - WSF - conducted most of the outreach to and recruitment of families 
between March and May 2004. Perhaps as many as 40,000 DC children were eligible for the scholarship 
program, based on data from the U.S. Census (Table ES-2). Inquiries about the program were made on 
behalf of almost 6,000 students, and nearly 2,700 applications were submitted during the recruitment 
period. A total of 1,848 applicants (69 percent of those who applied) provided all of the required 
documents and were deemed eligible for the program. Seventy-two percent of those eligible applicants 
were attending public school during 2003-04, whereas 28 percent were already attending private schools 
but met the eligibility requirements in the statute. 

Table ES-2. Number and Percentage of Participants, by Application Status: Spring 2004 













Public 


Private 


Measure 


Eligible 

Base 


Inquiries 


Applicants 


Eligible 

Applicants 


Eligible 

Applicants 


Eligible 

Applicants 


Number of 
students 


40,507 


5,863 


2,692 


1,848 


1,330 


518 


% of base 


100 


14 


7 


5 


NA 


NA 


% of inquiries 






46 


32 


NA 


NA 


% of applicants 






100 


69 


NA 


NA 


% of eligibles 








100 


72 


28 



NOTE: Because the eligible base, inquiries, and applicants included an unknown combination of public and private school 

students, it would not be appropriate to express the number of public or private eligibles as a percentage of those 
bases. 



SOURCE: Figure for the “Eligible Base” is based on data from the U.S. Census, population of the District of Columbia age 
5 to 17 under 185 percent of the federal poverty line in 2000. The exact number for 2004 is likely to differ 
somewhat from this 2000 figure. Figure for “Inquiries” provided by Fight For Children. Figures for 
“Applicants” and “Eligible Applicants” were drawn from the applicant database, with eligibility determined by 
Private School Aid Service (PSAS). Numbers of applicants from public or private schools were determined by 
cross-matching the name of the school each child was attending against a list of public and private schools in 
DC compiled from the DCPS web site and the NCES Common Core of Data. 



Scholarship and Placement Lotteries and Initial Use of Scholarships Awarded (Chapter 3) 

The program statute requires that scholarship recipients be randomly selected (e.g., by lottery), if the 
program or specific schools are “oversubscribed” — that is, have more demand for them than can be 
accommodated. The law also details congressional priorities to guide the award of scholarships and 
scarce seats to eligible applicants: (1) students attending a public school designated as in need of 
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improvement (SINI) under the No Child Left Behind Act at the time of application to the program, and 
(2) families that lack the resources to take advantage of the educational choices available to them. 

A total of 79 eligible applicants (4 percent) were from one of the 15 SINI-designated schools in spring 
2004 and were, therefore, given the highest priority in the lotteries. 1 An additional 1,251 eligible 
applicants (68 percent) were attending non-SINI public schools and were assigned the second-highest 
priority in the lotteries. The 518 eligible applicants (28 percent) from private schools were given the 
lowest priority, because they were considered to meet neither of the congressional priorities. These 
priority groups were used both to award scholarships and, later, to place scholarship recipients in the 
participating private schools of their choice. 



The Scholarship Lottery 



Figure ES -1 

Eligible Public School Applicants and Available 
Private School Slots, by Grade-Level Band: 
Spring 2004 and Fall 2004 



The first lottery was to distribute 
scholarships to eligible applicants. For 
public school applicants, each student’s 
probability of obtaining a scholarship was 
dependent not only on his or her 
membership in a priority group but also on 
the availability of new private school seats 
at various grade levels. The new seats in 
participating private schools were highly 
concentrated in the K-5 elementary grades, 
meaning there were more available seats in 
those grades than there were eligible 
applicants (Figure ES-1). Thus, in the 
lottery, K-5 students would receive 
scholarships automatically. The middle 
school grades (6-8) were modestly 
oversubscribed, and the high school grades (9-12) were severely oversubscribed. Only in those grades 
was there random assignment as part of the lottery. Since eligible private school applicants already held 
slots in their private schools, they were not constrained by slot availability in the same way as public 
school applicants. 




Total 



K-5 



6-8 

- Grade-Level Band - 



9-12 



1 While there were 79 applicants from the 15 schools designated as SINI in 2003, in advance of the first year lottery in May 
2004, DCPS designated an additional 73 schools as in need of improvement in August 2004, two months after the scholarship 
lottery (See http://silicon.kl2.dc.us/NCLB/index.asp . accessed February 15, 2004). A total of 535 program applicants came 
from 2003-designated or 2004-designated SINI schools, representing 29 percent of all program applicants this first year. Of 
these students, 433 (81 percent) were awarded scholarships. Twice as many Opportunity Scholarships were awarded to 
students applying from 2003- or 2004-designated SINI schools than were awarded to students applying from private schools. 
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Accounting for both the statutory 

priorities and the slot constraints within 
the grade-level bands, scholarship award 
probabilities were assigned to the various 
groups of eligible applicants and a 
custom-designed computer program 

awarded scholarships to students within 
each group (Figure ES-2). A total of 
1,366 scholarships were awarded in June 
2004, including the following: 

• All 79 applicants from SINI- 
designated schools. 2 

• All 772 non-SINI public school 
applicants in K to 5. 



Figure ES-2 

Scholarship Probability, by Type of Applicant and 
Grade-Level Band: Spring 2004 

Percent ioo 100 100 100 100 

100 

80 

60 

40 

20 

0 

Total K-5 6-8 9-12 

Grade-Level Band 

■ Public "SINI" □ Other Public B Private] 




• 255 (76 percent) of the non-SINI public school applicants in grades 6 to 8 and 44 (28 percent) of 

them in grades 9 to 12. 



• A total of 216 (43 percent) of applicants already attending private schools, with probabilities 
considerably lower (55 percent for those in grades K to 5, 42 percent for grades 6 to 8, and 17 
percent for grades 9 to 12) than public school applicants in all three grade bands. 



In sum, the scholarship lottery produced two groups of students for purposes of meeting statutory 
requirements for the DC Choice Opportunity Scholarship program evaluation. The 1,366 scholarship 
recipients will be the subject of annual performance reporting and comparison to DCPS nonapplicants. 
The 492 public school applicants who were entering grades subject to random assignment (grades 6-12) 
will contribute to the annual impact analysis: these include 299 students assigned to the treatment group 
(and also included in the performance reporting sample) and 193 students assigned to the control group. 
The 289 private school applicants who were not awarded scholarships belong to neither group. Since 
they previously attended and presumably will continue to attend private school using resources outside of 
the scholarship program, following these students after baseline would not contribute meaningfully to the 
evaluation. To conserve resources, this group of initial applicants will not be part of the evaluation going 
forward. 



However, the impact sample in the first year of the program is not, on its own, large enough for the 
evaluation to reliably draw conclusions about any differences in achievement outcomes that might be 
expected from an intervention of this kind. 2 Instead, the treatment and control groups from the first year 



' Because they represented such a small share of the overall applicant pool the first year and there was little possibility of 
separately analyzing impacts for this subgroup, all SINI applicants were awarded a scholarship. However, with more SINI 
schools identified for the next year, it is likely that a much higher number of students in this priority group will apply to the 
program in the future. Therefore, in succeeding years the group will be given highest priority but be part of the random 
assignment process that will enable that important category of students to be included in the rigorous impact evaluation. 

3 Evaluation sample size needs are calculated using statistical power analysis. It is an important part of evaluation design and 
demonstrates how well the design will be able to distinguish real impacts from chance differences between groups. The 
analysis takes into consideration such factors as the size of the impact expected, the proportion of students anticipated to be 
assigned to the treatment and control groups, and the likely attrition from the study’s data collection. The power analysis 
conducted for the DC Opportunity Scholarship Program suggests a target of 1,240 randomly assigned students, with slightly 
more than 800 students assigned to the treatment group and slightly more than 400 assigned to the control group, so long as the 
test score impacts in the outcomes in future years are at least moderately large (Appendix A). 
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lottery will be combined with those from the lottery for second year applicants, expected in April 2005, to 
provide a sufficient sample for the rigorous evaluation of program impacts. 



Placement Lottery and Follow Up 

After being notified of their scholarship offer, families were required to meet with officials at 
participating private schools and obtain conditional acceptance to the schools that they wanted their 
children to attend. Parents then submitted school preference forms to indicate and ra nk the top four 
private schools of their choice. These forms were used to place students, through a combination of a 
custom-designed computer placement lottery and followup case-by-case placements by WSF. There were 
a total of 1,366 scholarship winners: 

• A total of 1,040 were successfully placed, with the overwhelming majority of students placed in 
their most-preferred school. 

• The remaining 326 scholarship recipients did not complete the school search process and, 
therefore, could not be assigned a placement in a participating private school. 

Of the 1,040 who were placed, 1,027 had matriculated at their preferred private school by September 10, 
2004. This represents an overall initial scholarship usage rate of 75 percent. The usage rate for the 
impact sample’s treatment group is lower — 62 percent — because that group excludes students in grades 
K-5 and those who were already attending private schools when they applied to the program, subgroups 
that have significantly higher rates of scholarship use than do public school students in the middle and 
high school grades. 



Characteristics of Program Applicants (Chapter 4) 

In determining and interpreting program effectiveness later on, it is important to know how well the 
program is targeted to the disadvantaged families who are the focus of the program and, beyond that, 
what types of families and students apply, win scholarships in the lottery, and choose to use them to 
enroll in a private school. It is also useful to identify the extent to which public and private schools in the 
District are experiencing a significant loss or gain of students due to the first year of program 
implementation, because that information provides the foundation for our later examination of the impact 
of the program on DC schools. 



Public School Applicants Compared to Similar Public School Students in DC 

There are several reasons to examine the eligible public school applicants to the DC Opportunity 
Scholarship Program in relation to other DCPS students. Most clearly, the comparison provides a context 
for considering the kinds of students who might be attracted to the program in future years in the District 
or to a similar program in other locations. The income eligibility criteria for the program — family income 
within 185 percent of the federal poverty line — matches up quite closely with eligibility for the federal 
free or reduced-price lunch (FRL) program. Using this income indicator to compare public school 
program applicants to similarly low-income DCPS nonapplicants, we find some differences and 
similarities (Table ES-3): 
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Table ES-3. Characteristics of DC Public School Free or Reduced-Price Lunch 

Program Students, Program Applicants Versus Nonapplicants: Spring 2004 



Characteristic 


Applicants 1 


DCPS 

Sample 


Difference 


Baseline Test Scores 2 








Average Reading Percentile 


40.2 


36.4 


3 9 ** 


Percent missing 


25 


26 




Average Mathematics Percentile 


46.7 


43.0 




Percent missing 


25 


24 




Percent in Special Education 


17 


15 


1 


Percent missing 


24 


22 




Percent, by Race 








African American 


93 


88 


5** 


Hispanic 


6 


10 


-5** 


Other race 3 


2 


2 


0 


Percent missing 


0 


0 




Percent, by Gender 








Female 


51 


51 


0 


Percent missing 


0 


0 




Percent Participating in Free/Reduced- 
price Lunch Program 

Percent missing 

Sample size 


100 

0 

894 


100 

0 

44,740 


0 



* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 The sample size of 894 for the applicant group compared here differs from the total sample of 1343 public school applicants 
for two reasons. First, only 1,077 (80 percent) of the public school applicants could be identified conclusively in the DCPS 
accountability testing database, the only source of comparable data for both applicants and nonapplicants from DCPS. Most 
missing observations were in pre-K, first, or second grade, where accountability testing is optional. An additional 183 public 
school applicants in the accountability database were not enrolled in the free or reduced-price lunch (FRL) program. Because it 
is most analytically sound to compare similarly disadvantaged groups of applicants and DCPS nonapplicants, we included in this 
comparison only those students who had test scores in the database and who were confirmed eligible for FRL. To examine 
whether there was any substantial bias in our analysis, we compared the demographic characteristics of students who were tested 
by DCPS with those who were not tested, both within and across the applicant and nonapplicants samples, using a t test for 
difference of means to identify statistically significant differences. The testers did not differ significantly from the nontesters on 
any characteristic, except for grade, since testing is mandatory only in grades 3 and higher. Therefore, we are confident that 
tester/nontester bias does not affect the comparisons that we make here. 

2 Test-score results are in terns of National Percentile Ranks, with 50 as the median score. 

3 “Other race” includes students who were identified as white, Asian, American Indian, or Alaska Native. 

NOTE: Detail may not sum to totals because of rounding. Applicant sample includes all eligible applicants identified in the 

DCPS data base that were participating in the free- and reduced-price lunch program. 

SOURCE: Accountability testing database for DC public and charter schools, DCPS Office of 
Communications and Public Information. 
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• Program applicants scored somewhat higher on reading and mathematics accountability tests than 
the nonapplicants. 

• Applicants were more likely than nonapplicants to be African American and less likely to be 
Hispanic. 

• The two groups were similar regarding special education enrollment, gender, and enrollment in 
the FRL program (the latter by design). 



Similar patterns are evident when we use a subset of the applicants — just the program participants — to 
compare to economically disadvantaged DCPS students in the same grades, as the program statute 
requires for performance reporting. 4 



Applicants in the Impact and Non-Impact Samples 

While it is important to examine who applies to the DC Opportunity Scholarship Program, for the 
evaluation it is equally critical to assure that the applicants who will be the focus of the impact analysis — 
the applicants who were randomly assigned to treatment and control groups randomly assigned to 
treatment by the lottery — are similar prior to the beginning of the program. It is on this similarity 
between the groups, not only in characteristics easily measured but also in those not observed, that the 
scientific benefits of the randomized control trial (RCT) approach rests. We find no statistically 
significant differences between the two groups on any major educational or family background measures, 
confirming that the assignment lottery was conducted appropriately. 

It is also useful to review how the characteristics of the impact sample differ from the characteristics of 
students who applied but will not be included in the analysis of program effectiveness (the “non-impact 
sample” of applicants). The existence of many significant differences between the impact and non-impact 
samples limits extrapolation of the results of the impact analysis to characterize overall program impact. 
There are a few differences between the two groups. Compared to their non-impact sample counterparts, 
members of the impact sample: (1) scored higher in reading in grades 9 through 12, (2) are more likely to 
have a learning disability, and (3) are less likely to be of Hispanic ethnicity. 



Applicants by Type of Previous School 

The schools students previously attended may be associated with students’ educational or background 
characteristics, their parents’ attitudes, and ultimately the extent to which the program is effective for 
students seeking scholarships. Eligible applicants to the program came from four different types of 
schools. Four percent came from SINI-designated public schools, 54 percent from non-SINI regular 
public schools, 14 percent from public charter schools, and 28 percent from various private schools. The 
most important differences among the applicants include the following: 



4 The sample size of 894 for the applicant group compared here differs from the total sample of 1343 public school applicants 
for two reasons. First, only 1,077 (80 percent) of the public school applicants could be identified conclusively in the DCPS 
accountability testing database. Most missing observations were in pre-K, first, or second grade, where accountability testing 
is optional. An additional 183 public school applicants in the accountability database were not enrolled in the free or reduced- 
price lunch program. To keep the comparison balanced, they were excluded from the applicant group for purposes of this 
analysis. 
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• Among high school applicants, SIN1 public applicants scored lower in reading, whereas non-SINI 
public applicants scored higher. 5 

• The average family income of all applicants was $18,742, with SINI and non-SINI public 
applicants reporting somewhat lower incomes and private school applicants reporting somewhat 
higher incomes. 

• The mothers of applicants reported an average of almost 13 years of formal education. The 
mothers of non-SINI public applicants reported slightly lower levels of education, and the 
mothers of private school applicants slightly higher levels. 



Applicant Response Rates Among Public and Private Schools 

A central question in the debate surrounding school choice — and one of the topics the statute requires the 
evaluation to address — is whether a scholarship program has an impact on the larger public and private 
school systems. Such “systemic effects” could take place if significant percentages of students in the 
public school system or in specific schools apply for, receive, and use scholarships to transfer to private 
schools. With regard to the public schools, the DC Opportunity Scholarship Program could have either 
positive or negative effects. One theoretical argument suggests that scholarship programs will divert 
funding and the most motivated students from public schools to private schools, leaving the public school 
system with fewer resources with which to educate the remaining student population. Another theory is 
that schools behave in a manner similar to firms and will respond to competition by becoming more 
efficient. In the case of schools, it is the risk of losing students and subsequently funding that may 
provide an impetus for public schools to provide better services and produce better student outcomes. 
Private schools in jurisdictions with greater school choice may face similar incentives to improve or 
expand in order to retain as many of their current students and attract more. 

To provide the basis for a later analysis of the effects of the DC Opportunity Scholarship Program on 
schools, we first need to describe the extent to which public schools have so far been affected by program 
applications and scholarship users — a possible predictor of the level of competitive pressure the public 
schools may experience. The school-level scholarship application and use rates this first year suggest that 
relatively few public schools have experienced a significant loss of students as a result of the scholarship 
program (Table ES-4): 

• Over one-quarter of the public schools in the District experienced no student losses due to the 
program. 

• Another 56 percent of DCPS schools had program-related transfers out that totaled less than 2 
percent of their student populations. 

• Seventeen percent of District schools lost about 2 to 4 percent of their students. 

• Finally, 2 percent of the public schools in the District experienced more significant student 
transfers of over 4 percent under the program. 



5 Test scores were not available for the private school applicants. 
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Table ES-4. Public School-Level Scholarship Application and Usage Rates: 
Spring 2004 and Fall 2004 



Percent of 
Student Body 


Applied for the Program 


Used a Scholarship 


Number of 
Schools 


Percent of 
Schools 


Number of 
Schools 


Percent of 
Schools 


0 


25 


13 


51 


26 


O 

1 

o 


47 


24 


63 


32 


1.1 -2.0 


56 


28 


47 


24 


2.1 -3.0 


38 


19 


21 


11 


3.1 -4.0 


12 


6 


12 


6 


4.1 - 


19 


11 


3 


2 


Total 


197 


100 


197 


100 



NOTE: Detail may not sum to totals because of rounding. 



SOURCES: Application and usage numbers by school generated from the Applicant Database and 
Washington Scholarship Fund (WSF) Placement Database. Enrollment figures for 
DCPS are from the 2002-03 school year and were obtained from Membership in the 
District of Columbia Public Schools by School and Grade, October 7, 2003, available 
on the DCPS web site, www.kl2.dc.us/dcps/data/enrollment/membership- 
Oct.703_official_.pdf. Enrollment figures for DC public charter schools chartered by 
the District Board of Education are from the 2002-03 school year and were obtained 
from the Common Core of Data, National Center for Education Statistics. Enrollment 
figures for the DC public charter schools chartered by the DC Public Charter School 
Board are from the 2003-04 school year and were obtained from the Board’s web site 
at www.dcpubliccharter.com. 

In contrast, a similar analysis of the participating private schools suggests that students using DC 
Opportunity Scholarships make up a significant share of their enrollments. In more than one-quarter of 
those private schools, nearly 20 percent of their students are using Opportunity Scholarships; in another 
37 percent, scholarship students make up between 5 and 20 percent of their student populations in 2004- 
05. 
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1. INTRODUCTION 



The District of Columbia School Choice Incentive Act of 2003 was passed by Congress in January 2004. 
The Act provided funds for District of Columbia Public School (DCPS) improvement activities and 
charter school facility acquisitions. Most notably, the statute established what is now called the DC 
Opportunity Scholarship Program— the first federal government initiative to provide K-12 education 
scholarships, or vouchers, to families to send their children to private schools of choice. 

The statute requires that this 5 -year scholarship program be rigorously evaluated by an independent 
research team. This document is the first of a series of annual reports from that team, as mandated by 
Congress. The report describes the purposes and design of the scholarship program, the first-year 
implementation activities that generated 1,848 eligible applicants and 58 participating private schools, the 
process of randomly awarding scholarships to 1,366 student applicants, and the characteristics of both 
applicants and scholarship users. The report provides an important foundation for the later examination of 
program impacts. 



1.1 District of Columbia School Choice Incentive Act of 2003 

In January 2004, the U.S. Congress passed the DC School Choice Incentive Act of 2003, Title III of 
Division C of the Consolidated Appropriations Act, 2004, P.L. 108-199. The statute established a new, 5- 
year school choice program for low-income residents of Washington, DC. The key elements of the 
program include the following: 

• To be eligible, students entering grades K-12 must reside in the District and have a family income 
at or below 185 percent of the federal poverty line. 

• Participating students will receive scholarships of up to $7,500 to cover the costs of tuition, 
school fees, and transportation to a participating private school of choice. 

• Scholarships are renewable for up to 5 years (as funds are appropriated), as long as students 
remain eligible for the program and remain in good academic standing at the private schools they 
are attending. 

• If there are more eligible applicants than available scholarships or open slots in private schools, 
applicants are to be awarded scholarships and admission to private schools by random selection 
(e.g., by a lottery). 

• Private schools participating in the program must be located in Washington, DC, and agree to 
program requirements regarding nondiscrimination in admissions, fiscal accountability, and 
cooperation with the evaluation. 

Certain groups of students have priority in obtaining access to the program: (1) those coming from public 
schools identified as in need of improvement (SINI) under the federal Elementary and Secondary 
Education Act and (2) those whose families lack the financial resources to take advantage of available 
educational options. 

The law also charged the U.S. Secretary of Education and the Mayor of the District of Columbia with 
selecting both a program implementer and an independent evaluator of the program. The Secretary 
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designated the Office of Innovation and Improvement (Oil) within the U.S. Department of Education 
(ED) as the lead agency for funding and monitoring the implementation of the program, and the Institute 
of Education Sciences (IES) to take the lead in funding and monitoring the independent evaluation. 
Under current appropriation levels — about $13 million annually — the program is likely to support 1,800 
to 2,000 scholarships, depending on the tuition levels of the selected private schools. 



1.2 Congressionally Mandated Evaluation 

Section 309 of the Act describes the requirements for an independent evaluation of the DC Opportunity 
Scholarship Program. The Secretary of Education is to ensure the following: 

• “The evaluation is conducted using the strongest possible research design for determining the 
effectiveness” of the school choice program; 

• The results of the evaluation regarding the impact of the program on the participating students 
and nonparticipating students and schools in the District are disseminated widely. 

Early on, IES determined that the foundation of the DC Opportunity Scholarship Program evaluation 
would be a randomized controlled trial (RCT), comparing outcomes of eligible applicants (students and 
their parents) randomly assigned to receive or not receive a scholarship. 6 This decision was based on the 
mandate to use rigorous evaluation methods, the expectation that there would be more applicants than 
funds and private school spaces available, and the requirement to use random selection to determine who 
receives a scholarship. In addition, the law clearly specified that such a comparison in outcomes be 
made. 7 This component represents the impact analysis and will provide evidence on the effectiveness of 
the program. 8 

The law also called for the evaluation to track program progress in other ways. For example, the 
evaluation must compare students participating in the scholarship program to students in the same grades 
in the DC Public Schools. However, DCPS students who did not apply to the scholarship program are 
likely to be quite different from those who applied and are participating — in ways we can observe and 
ways we cannot. Comparing outcomes between participants and nonapplicants is, therefore, not a reliable 
measure of program effects. Instead, this type of performance reporting will be combined with other 
data collection and analysis that examines the context in which the program is operating. 

In spring 2004, IES initiated a competitive bidding process to select an initial technical advising team as 
well as an entity to design and implement the 5-year impact evaluation of the program. The technical 



RCTs are commonly referred to as the “gold standard” for evaluating educational interventions; when mere chance determines 
which eligible applicants receive access to school choice, the students who apply but are not admitted comprise an ideal 
“control group” for comparison with the school choice “treatment group.” Both groups of participants are equally motivated to 
obtain new educational options, and nothing except a random draw distinguishes those who receive the opportunity from those 
who do not. Therefore, any differences in the two groups in subsequent years can be attributed to the impact of the program. 
In contrast, the results of school choice studies that are not based on RCTs must be interpreted and used more cautiously, 
because comparisons between the applicants and a group of students who chose not to apply will likely reflect not only the 
impact of the program but also differences between the groups in motivation and other unmeasured characteristics. See below 
for more detail on the RCT analysis approach. 

7 See 309 (a)(4)(A)(ii). 

8 The RCT approach was also used by researchers conducting impact evaluations of the New York City, Dayton, Ohio, and 
Washington, DC, private scholarship programs. 
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advising contract was competed quickly, so that a group of experts would be in place to advise ED and 
the program operator regarding the baseline data collection and the lotteries that are essential to both the 
effective launch of the program and the subsequent impact evaluation. In March 2004, the technical 
advising contract was awarded to a research consortium led by Westat and including Georgetown 
University and Chesapeake Research Associates. Later, in July 2004, the competition for the 5-year 
impact evaluation contract concluded with an award to the Westat-Georgetown-Chesapeake team. 



Research Questions 

Based on guidance in the statute, the research team plans to conduct a comprehensive and rigorous RCT 
evaluation of the impact of the scholarship program on participating students and families. Specifically, 
the impact analysis will address the following research questions: 

1. What is the impact of the program on student academic achievement? The law places high 
priority on examining whether the program — the availability and offer of scholarships — improves 
the academic achievement of eligible students. This question can be addressed most rigorously by 
comparing the academic achievement of student applicants randomly assigned by a lottery to 
receive and not receive scholarships. 

2. What is the impact of attending private versus public schools? Because some students offered 
scholarships will choose not to use them, the research team will use accepted statistical methods 
to examine the effects for students who take the scholarship offer and successfully enroll in a 
private school. 

3. What is the impact of the program on other student measures? The law calls for examining 
other indicators of student and school success, including persistence, retention, graduation, and, if 
possible, college enrollment. In addition, the legislation requires the evaluation to assess the 
school safety of students who receive the scholarships relative to those who did not receive 
scholarships. 

4. What effect does the program have on student and parent satisfaction with the educational 
options available in the District and with children’s actual school experiences? A key desired 
outcome of scholarship programs is an increase in both the school choices possible and parents’ 
and students’ satisfaction with the choices they have made. These issues will be examined by 
comparing the satisfaction and reasons for applying to the DC Opportunity Scholarship Program 
among applicants assigned by lottery to receive scholarships and those assigned to not receive 
scholarships. 

The evaluation will also address other issues posed in the law through program performance analysis: 

5. How well are scholarship recipients performing relative to students in DCPS? As noted earlier, 
the law asks for a comparison of the academic achievement of students who participate in the 
program with the achievement of their grade level counterparts in the DCPS. 9 



9 Because answering this question requires analysts to deviate from the random assignment design, and the testing conditions for 
scholarship students and DCPS students will differ, inferences from the results of this specific comparison will be drawn with 
caution. 
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6. To what extent is the program influencing public schools and expanding choice options for 
parents in Washington, DC? Scholarship programs have been hypothesized to affect not only 
the students who receive the scholarships but also the broader population of public schools and 
students. Theory suggests that these broader outcomes could occur if a significant number of 
students move from public to private schools. The public schools might experience a reduction in 
per student funding that affects their offerings, a change in average student performance, or they 
may respond to the competition for students by changing curricula, adopting new themes or 
missions, and modifying existing policies and practices to make the public schools more 
attractive to students with schooling options. Choice programs might also affect the larger 
population of private schools, beyond those in which the programs’ participants are currently 
enrolled; if choice programs are successful, additional private schools may choose to participate, 
new schools may be established to meet enrollment demand, or existing schools might expand 
capacity. 



Data Collection 

To answer these questions, the evaluation will draw on different types of data — some available from 
DCPS, some collected for the purposes of this study. These data will include preprogram (baseline) 
measures of family background and student achievement. The baseline measures allow us to verify that 
students randomly assigned to the scholarship and nonscholarship groups were, in fact, similar before the 
program; the measures also enable us to create subgroups of students whose impacts we might want to 
examine separately, such as students with low prior achievement. Additional data collected will include 
annual “in program” measures (e.g., parents’ satisfaction with their children’s school, students’ academic 
achievement), which will serve as outcomes for the rigorous evaluation of program impacts (see 
Table 1-1) 



Impact Analysis 

It is well known that the independent effects of school choice on student outcomes are difficult to 
estimate. Perhaps the most significant difficulty faced by researchers is selection bias— the self-selection 
of families to even seek out a new school choice for their child, and the mutual student/school decision 
process that selects students into different types of schools. Because this bias is generally a result of 
unmeasurable factors, most researchers have preferred the use of an RCT to a dependence on non- 
experimental (nonrandomized) statistical methods. Since the DC Opportunity Scholarship Program 
provides for the random distribution of scholarships through a lottery, we will, therefore, use RCT 
methods to estimate most program impacts. 10 



10 For this report, which is descriptive, as well as for the later impact analysis reports, we will use several tests for calculating 
statistical significance, or the level of confidence that evaluators have that a difference between groups did not occur merely by 
chance. For most of the comparisons that we make, we use the “Student’s t test.” The t test is commonly used when the factor 
being considered, such as test scores, tends to be distributed on a normal, bell-shaped curve. Unlike some significance tests, 
the t test incorporates information about the distribution of values in both comparison groups, and not just the overall 
population, and thus is a more precise measure of statistical significance than the Z test, for example (see Russell A. Langley, 
Practical Statistics Simply Explained (New York: Dover, 1970), 160-165. When we compare the characteristics of one 
specific group to another specific group, the application of the t test is conventional and straightforward. When we compare 
the characteristics of one specific group with several other specific groups, as in the comparisons of applicants from various 
types of schools in Chapter 4, we apply the t test to calculate the significance of the difference between the average score for 
the group in question and the average score for all of the other groups combined. When the characteristic in question is not 
normally distributed — such as gender, which is an either/or and not a more-or-less — we use the “chi-squared” test of statistical 
significance. 
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Table 1-1. 



Data Sources 



Data Source 


Description 


Student assessments 


• Baseline, or preprogram measures of student achievement for public school 
applicants will come primarily from the SAT-9 standardized assessment 
administered by DCPS as part of their spring testing program. 1 

• Each spring after the baseline year, the study will administer the SAT-9 to 
all students who were offered scholarships as well as all members of the 
control group who did not receive a scholarship 

• DCPS test score data will be obtained for all public school students in those 
years, to compare with the students participating in the program. 


Parent surveys 


• Surveys of parents (of all applicants) will be conducted in all 4 years of 
impact evaluation data collection. 

• Topics will include reasons for applying, satisfaction with school choices, 
and perceptions of school safety, educational climate, and offerings. 


Student surveys 


• Each year after baseline, surveys will be conducted with all applicants who 
are in grades four and higher. 

• Topics will include students’ satisfaction with their schools, perceptions of 
safety, and other characteristics of their school program and environment. 


Principal surveys 


• Surveys will be conducted each year of principals of all 109 private schools 
in DC and principals of all of the 1 60 regular public and charter schools in 
DCPS. 

• Topics will include school organization, safety, climate, principals’ 
awareness of and response to the DC Opportunity Scholarship Program, 
and, for private school principals, why they are/are not participating. 



Baseline achievement will be collected only for applicants from public schools because, as described in the Impact Analysis 
Sample section, applicants who were already attending private schools will not be included in the impact analysis. For 
public school applicants who did not participate in regular DCPS testing in the year they applied to the program (e.g., 
particularly children below grade 3), the study will administer the equivalent DCPS assessment to these students in the fall 
after application. All other data will be collected for all applicants, both from public and private schools. 



Impact Analysis Sample 

The RCT approach rests on random assignment or, in the case of the DC Opportunity Scholarship 
Program evaluation, a lottery to create two statistically equivalent groups of students from among 
program applicants: (1) a “treatment” group that receives a scholarship, and (2) a “control” group that 
does not receive a scholarship. Because the two groups are generated from the same pool of applicants, 
they are equally likely to be motivated to participate in the program and to reap any benefits from it. And 
as long as the pool of applicants is sufficiently large, the random assignment of students into treatment 
and control groups should produce groups that are similar in other characteristics, both those we can 
observe and measure (e.g., family income, prior academic achievement) and those we cannot (e.g., 
motivation to succeed). The random assignment assures that all observed and unobserved characteristics 
are equally represented in both groups. 
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However, according to the statute, the random assignment that is the means to create the treatment and 
control groups can only be used to help allocate scholarships under particular circumstances. Finally, it 
may not be appropriate to include some student applicants in the impact evaluation, even if they 
participate in the lottery. As a result of all of these conditions, the impact analysis sample will perform 
the following: 

• Exclude applicants who are already attending private schools. The statute contained no provision 
to exclude from the program students who were currently enrolled in private schools but 
otherwise eligible to participate. 11 A substantial number of private school students did apply to 
the program, as described in Chapters 2 and 3. However, because those students intended to use 
the DC Opportunity Scholarship to continue to attend private schools, measuring the difference in 
outcomes between private school applicants who did and did not receive a scholarship through 
the lottery would likely only answer the question of whether a different type or amount of 
scholarship funds affect student outcomes. While that question is of some policy interest, it is not 
the main focus of the evaluation as specified in the legislation. Therefore, applicants currently 
enrolled in private schools will not be part of the impact analysis sample. 

• Include only public school applicants in grades where there are more applications than there are 
available private school slots. A lottery is a fair and efficient way of distributing scholarships 
when there are too many applicants, but the law specifies that random selection be used only 
when the program or particular grades are oversubscribed. 

• Exclude public school students who automatically receive a scholarship. It is possible that even if 
many students apply to participate in the program, there may not be oversubscription (i.e., more 
demand than slots) in some grade levels. In those grade levels, all applicants will receive 
scholarships and there will be no control group. 

Thus, the impact evaluation of the DC Opportunity Scholarship Program depends on the extent to which 
large numbers of eligible DC families with public school students apply to the program. The treatment 
and control groups must be of a sufficient size to allow us to detect and measure any difference in 
outcomes between the two groups (the “impact”) with statistical certainty. A procedure called “power 
analysis” is used to determine the sample sizes necessary to enable the study to answer the central research 
questions and to measure program effects that are large enough to be both meaningful in students’ lives and 
relevant to policy debates about the efficacy of school choice interventions. The power calculations 
conducted for the evaluation suggest that a total of at least 1 ,240 public school applicants must be randomly 
assigned, with over 800 assigned to the treatment (scholarship groups) and over 400 assigned to the control 
(non-scholarship) group in order for the impact analysis to be able to detect moderately large test score 
effects (see Appendix A for more detail). 



11 Some of these applicants from private schools were already relying on scholarship funds in order to attend those schools. 
However, the scholarships they were receiving may have been less generous than those available under the federal DC 
Opportunity Scholarship Program. 
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General Statistical Approach: Estimating the Impact of the Offer of a Scholarship 

Given appropriately sized treatment and control groups, the strategy for analyzing impacts is well 
established. To motivate the discussion of how we identify the effect of the scholarship program on test 
scores, it is useful to begin with a simple representation of the selection problem as a missing data 
problem, using the potential outcomes approach. This approach defines causal effects in terms of 
potential outcomes or counterf actuals. Conceptually, the causal effect of treatment — the scholarship — is 
defined as the difference between the “outcome for individuals assigned to the treatment group” and the 
“outcome for the treatment group if it had not received the treatment,” or: 

(E.l) “E(Yi| X„ Ti =1)” - “E(Y; |X i5 T, =0)” 

In the case of scholarships, the treatment effect— the effect of the scholarships on academic 
achievement— would be defined as the difference between “test scores for program students” and “test 
scores for program students if they had not received a scholarship.” The fundamental problem is that a 
student is never observed simultaneously in both states of the world. What is observed is a student in the 
treatment group (Ti =1) or in the control group (T =0). The outcome in the absence of treatment, E( Y; |Xj, 
Tj =0), is then the counterfactual— what would have occurred to those students receiving the scholarships 
if they had not received them. 

If students receiving scholarships were identical to other students in both observable and unobservable 
characteristics, the counterfactual could be generated directly from an appropriately selected comparison 
group. Valid comparison groups are rarely found in practice, however. The random assignment of 
students into the program generates the counterfactual from the control group— eligible applicants who did 
not receive a scholarship. 12 If correctly implemented, random assignment yields statistically equivalent 
groups and allows estimation of the program impact through differences in mean outcomes between the 
two groups. 

Consistent with this approach is the following basic analytic model of the effects of school choice 
scholarships on outcomes. Consider first the outcome equation for the test score of student i in year t. It 
is reasonable to assume that test scores (Y it ) are determined as follows: 

(E.2) Y^ =»■+ x Tj t + X, y+ £ it if t>k (period after program takes effect) 

Equation (E.2) estimates the effect of the offer of a scholarship on student outcomes. Under this model, 
commonly referred to as the “Intent to Treat” (ITT) estimation, all students who were randomly assigned 
by virtue of the lottery are included in the analysis, regardless of whether a member of the treatment 
group uses the scholarship to attend a private school. In E.2, T it is equal to one if the student has the 
opportunity to participate in the scholarship program (i.e., the award rather than the actual use of the 
scholarship) and equal to zero otherwise. X; is a vector of student characteristics (measured at baseline) 
known to influence future academic achievement, such as prior test scores, mother’s level of education, 
family income, etc. In this model, x represents the effect of scholarships on test scores for students in the 
program, conditional on X;. With a properly designed RCT, using a concise and judiciously chosen set of 



12 See the following studies, which all use the same data from an evaluation of a New York City privately funded scholarship 
program: William G. Howell, Patrick J. Wolf, David E. Campbell, and Paul E. Peterson, “School Vouchers and Academic 
Perfonnance: Results from Three Randomized Field Trials,” Journal of Policy Analysis and Management 21 (2000): 2; John 
Barnard, Constantine E. Frangakis, Jennifer L. Hill, and Donald B. Rubin, “Principal Stratification Approach to Broken 
Randomized Experiments: A Case Study of School Choice Vouchers in New York City,” Journal of the American Statistical 
Association 98 (2003): 462; Alan B. Krueger and Pei Zhu, Another Look at the New York City School Voucher Experiment, 
Working Paper Series, Education Research, (Princeton, NJ: Princeton University, March 2003). 
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statistical controls for characteristics that predict future achievement should improve the precision of the 
estimated impact. 13 That treatment effect, x, should be identical to the difference in mean outcomes 
between the treatment and the control groups. 

Since the initial applicants were randomized within certain relevant subgroups, as described in Chapter 3, 
we will analyze program impacts using a randomized block design. We are interested in how academic 
achievement (Y) is affected by the assignment into the scholarship program within each block (B) or 
group of size n. The impacts are then estimated as: 

(E.3) Y ikt = p+ x Tfct +X b j =2 pj B lk + X ik y+ e iM 

where 

i = 1, ,n observations and k=l,....,b blocks(defmed by grade and priority status); 

Yjj is the outcome for student i in block j, at time t; 
p is the overall mean outcome (e.g., test score); 
x is the treatment (scholarship program) effect; 

Pj is the j th block effect; 

T it is assignment into the scholarship program; 

Bji is the block assignment; 

Xjj represents observable characteristics, measured at baseline; and 
Sjj is the random error; independent, N(0,o i: 2 ). 

This analytical framework follows naturally from the group randomization and is easily implemented and 
interpreted. Y can be measured in several different dimensions, including test scores, school satisfaction, 
parental satisfaction, grade completion, including where appropriate, high school graduation, etc. p is 
average outcome for all program members, pj is the average block effect, and x is the effect of 
scholarships on academic achievement. 14 



Estimating the Impact of the Use of Scholarships 

Even with a properly implemented RCT, we may expect that not all applicants placed by random 
assignment into the treatment (scholarship offer) group will actually use the scholarship at a private 
school. That is, some scholarship recipients may choose not to use their scholarship and instead attend a 
public school. This type of non-participation or underutilization of treatment services has been observed 
across all RCT settings, including medical trials, job training and health insurance experiments, as well as 
in previous school scholarship RCTs such as the one of the Milwaukee Parental Choice Program. 



13 For a spirited debate about the use of this technique in the context of school choice research, see William G. Howell and Paul 
E. Peterson, “Uses of Theory in Randomized Field Trials: Lessons from School Voucher Research on Disaggregation, Missing 
Data, and the Generalization of Findings,” American Behavioral Scientist 47 (Jan. 2004): 634-657; Krueger and Zhu, “Another 
Look , ” 658-698; Paul E. Peterson and William G. Howell, “Efficiency, Bias, and Classification Schemes: A Response to 
Krueger, A.B. and Zhu, P., ‘Another Look at the New York City School Voucher Experiment,’” Working Paper Series, 
Education Research (Princeton, NJ: Princeton University, March 2003), 699-717; Alan B. Kiuger and Pei Zhu, “Inefficiency, 
Subsample Selection Bias, and Nonrobustness: A Response to Peterson, P.E. and Howell, W.G., ‘Another Look at the New 
York City School Voucher Experiment,’” Working Paper Series, Education Research (Princeton, NJ: Princeton University, 
March 2003): 718-728; Paul E. Peterson and William G. Howell, “Voucher Research Controversy: New Looks at the New 
York City Evaluation,” Education Next A (Spring 2004): 73-78. 

14 Depending on the extent to which the randomly assigned applicants are clustered in their schools, some adjustments to the 
standard error estimates may be necessary. 
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Policymakers are typically interested in the effect of scholarship use on student achievement, in addition 
to the offer of the scholarship. To estimate the impact, we will use a model commonly referred to as the 
“Impact of the Treated” (IOT), which statistically estimates the impact of actual scholarship use. 
Instrumental variable analysis provides us with a well-established method to generate an unbiased 
estimate of the scholarship impact on the treated from the ITT estimator. 15 



Performance Reporting Analysis 

To fulfill the requirements specified in the law, we will compare the outcomes of all participating 
students with the outcomes of similar DCPS students who did not apply to the program. 



Performance Reporting Sample 

The group of student applicants that will be examined as part of performance reporting differs from the 
impact analysis sample in several important ways. The impact analysis sample includes all public school 
applicants who were randomly assigned to receive or not receive a scholarship as part of the lottery (i.e., 
students in grade bands for which there were more applicants than there were private school slots 
available). In contrast, the performance reporting sample performs the following: 

• Excludes students who did not receive a scholarship as part of the lottery (the control group). 

• Includes students who both received a scholarship automatically and those who received one 
through the lottery (treatment group) 

• Includes scholarship recipients who were already attending private schools at the time of 
application. 

• Will focus on students who chose to use their scholarship; while the law does not define what it 
means to “participate,” performance reporting in the evaluation’s later reports will examine the 
differences between students who were given the option of participating (i.e., received a 
scholarship) and those who exercised that option (i.e., who used the scholarship to attend a 
private school). Both groups will be compared to DCPS students in the same grades who did not 
apply to the DC Opportunity Scholarship Program. 



Statistical Approach 

The performance reporting comparisons will focus on student achievement, both as specified in the 
statute and because that is the only measure that will be similar for DCPS students and those participating 
in private schools through the DC Opportunity Scholarship program. 16 In order to ensure comparability 
in student assessment, the evaluation will make every effort to administer the same test to program 



15 For an extended discussion of the use of this technique under such circumstances, see William G. Howell and Paul E. Peterson, 
with Patrick J. Wolf and David E. Campbell, The Education Gap: Vouchers and Urban Schools (Washington, DC: Brookings 
Institution Press, 2002): 49-51. 

16 That is, there are no readily available measures of student safety or parent satisfaction for students in DCPS that are 
comparable to those being collected from program participants. 
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participants that is used by DCPS. DCPS will provide the evaluation team with test score and 
background data on public school students. 

The analysis will be conducted by comparing the mean test scores of program participants and DCPS 
nonapplicants, testing for the statistical significance of the difference. To create the most relevant group 
of DCPS students for comparison, we will draw from the DCPS database the group of non-applicant 
students who qualify for the program (i.e., eligible for free/reduced-price lunch), stratified by grade level 
to match our scholarship performance reporting sample. We will present these comparative results as 
descriptive findings, since the absence of random-assignment to the scholarship or public school 
conditions would render any causal claims highly speculative. 



Reports 

The law requires the Secretary to submit to the Congress annual reports resulting from the independent 
evaluation by December 1 each year and a final report not more than a year after the 5-year program ends. 
These reports should provide the Congress, other policymakers, the research community, and the public at 
large with important new information about what happens to students, families, schools, and communities 
when educational options are expanded for urban low-income families through public policy. 



1.3 Organization of This Report 

This report is the first in the series of required evaluation reports to Congress. While the focus of the 
evaluation is on examining the effectiveness of the program, no impact information is available at this 
point because the initial cohort of program participants — those who applied in spring 2004 to receive 
scholarships for the 2004-05 school year — just recently matriculated at their new schools. Instead, this 
report examines the extent of student and school interest in the program and the characteristics of those 
participating. 

• Chapter 2 describes the recruitment and application activities that resulted in the submission of 
2,692 initial applications for the program, of which 1,848 were deemed eligible. It also describes 
the characteristics of the group of 58 private schools in the District that agreed to participate in 
the first year of the program. 

• Chapter 3 presents information about the lotteries that determined which eligible applicants 
would receive scholarships and which scholarship students would be assigned seats when 
individual schools were oversubscribed. 

• Chapter 4 describes the characteristics of program applicants, including their average test scores, 
family background, parental involvement in their education, type of school they attended 
previously, and parental assessments of their previous school. It also presents initial data 
regarding the extent to which DC schools are exposed to greater competition for students as a 
result of program implementation. 
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2. PROGRAM IMPLEMENTATION IN 2004: RECRUITMENT AND APPLICATIONS 



Any evaluation, particularly a randomized control trial (RCT), must take into consideration the number 
and flow of program applicants and participating institutions. In the case of the DC Opportunity 
Scholarship Program evaluation, the design was predicated on attracting a sufficient number of applicants 
to be able to create two sizable and randomly assigned groups of students whose outcomes could be 
compared: (1) eligible applicants who receive scholarships to attend participating private schools as part 
of the lottery and (2) eligible applicants who do not receive scholarships by virtue of the lottery. 

The recruitment and application period for the program began at the end of March 2004, immediately 
after a partnership led by the Washington Scholarship Fund (WSF) was selected to implement the DC 
Opportunity Scholarship Program, and continued throughout the spring and early summer of 2004. The 
process produced agreements from 58 private schools in the District to participate in the federal program 
during the first year and applications from 2,692 students seeking scholarships, of whom 1,848 were 
eligible. This chapter describes in more detail the activities and results of these efforts to recruit schools 
and students to participate in the program. 



2.1 Schools 

An initial task of the WSF-led implementation team was to recruit DC private schools to participate in the 
program during the first year. The team directly contacted each of the 109 private schools in the District 
and also worked closely with umbrella organizations that represent groups of schools, such as the 
Catholic Archdiocese of Washington, the Council for American Private Education, and the Association of 
Christian Schools International. The team also hosted an information session for private school officials 
in April. Several representatives of independent private schools at that meeting said that their admissions 
process for 2004-05 had already 
concluded and their schools were fully 
enrolled. However, some of them 
expressed an interest in participating in 
the program anyway, and others 
suggested that they would participate in 
the future if the application and 
admissions schedule could be moved to 
earlier in the school year. 

Despite the challenges stemming from the 
late start of the program, the 
implementers recruited 58 DC private 
schools to participate in the program in 
2004-05, comprising just over one-half of 
the private schools in the District (Figure 
2-1 and Table 2-1). 17 



Figure 2-1 

DC Private School Participation in the Program: 
2004-05 



Schools Participating 
by Accepting 
Scholarships Only for 
Pre-admitted 
Students 



Schools Participating 
by Creating Slots for 
DC Choice 
50% 




Note: Percents may not aedd to 100 due to rounding. 



17 To participate in the program, schools are required to sign a formal letter of agreement that requires them to (1) submit an 
independent financial and management audit to WSF annually to document financial stability, (2) provide a Certificate of 
Occupancy, (3) document that the school is operating lawfully in the District and is in compliance with DC health, safety, and 
fire codes, and (4) agree to abide by all applicable nondiscrimination laws. 
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Table 2-1. Number of DC Private Schools Participating in the DC Opportunity Scholarship 
Program: 2004-05 



Private Schools 


Number of 
Schools 


Percent 


In the District of Columbia 


109 


100 


Participating in the program in some capacity 


58 


53 


Set aside slots for DC Opportunity Scholarship Program 
participants 


54 


50 


Agreed to accept scholarships only for eligible students 
already admitted 


4 


4 



NOTE: Detail may not sum to totals because of rounding. 

SOURCE: “School Directory, D.C. K-12 Scholarship Program, 2004-05 School Year,” Washington Scholarship Fund, 
June 2004. 



Fifty-four of the 58 schools made new slots available to scholarship students; the remaining four schools 
had already closed their admissions and were only willing to accept new students awarded Opportunity 
Scholarships who already had been accepted to those schools during their regular admissions period. This 
small subgroup of scholarship winners that we describe as private school “pre-admits” were attending 
public schools in 2003-04 but obtained acceptance to a private school independent of and prior to the 
launch of the scholarship program. Some data were obtained regarding 53 of the 58 participating 
schools. 18 



Religious Affiliation, Location, and Students Sensed by Participating Schools 

The private schools that chose to participate in the DC Opportunity Scholarship Program in the first year 
are a diverse group but have some similar characteristics. Most are religiously affiliated and have been 
long established in the area. About half (27 of the 52 schools that reported their affiliation, or 5 1 percent) 
are affiliated with the Catholic Church; 1 1 (2 1 percent) are affiliated with a non-Catholic religion; and 1 5 
(28 percent) are independent nonsectarian schools. More than three-quarters of the schools were 
established before 1983, with a large share of them founded before 1955. The most recently established 
private school to participate in the program opened in 2002. 

The private schools that are participating in the DC Opportunity Scholarship Program are located in every 
ward of the District. The highest concentration of participating schools, nearly one-quarter of them, are in 
Ward 4. The remaining participating schools are almost evenly distributed throughout the other seven 
wards of the city. 19 

Most of the participating private schools already are serving a high proportion of students of color 
(Table 2-2). On average among the schools, 82 percent of their students in the current year are African 
American, Latino, Asian, or Native American, compared to 95 percent average minority populations in 
regular DCPS schools. Sixty percent of the participating private schools have student bodies that are 



18 WSF surveyed the participating schools about their characteristics and offerings in order to provide applicants and scholarship 
winners with information they could use in seeking a school placement. Five schools did not respond to the survey. 

19 The ward location of one participating school that provided data was not available. 
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entirely from minority racial/ethnic groups, but a few schools serve fewer than 10 percent of students 
from those groups. 

Table 2-2. Enrollment, Race/Ethnicity, and Student- Teacher Ratio of Students Served by 
Participating Private Schools Compared with Regular DCPS Schools: 2003-04 



Characteristic 


Average 


Highest 


Lowest 


Valid N 1 


Enrollment (number of students) 










Private schools 


206** 


1,056 


12 


52 


DCPS schools 2 


414 


1,442 


127 


139 


Percentage of students from racial/ 
ethnic minority groups 










Private schools 


82** 


100 


8 


52 


DCPS schools 


95 


100 


29 


139 


Average student-teacher ratio 3 










Private schools 


11:1** 


20:1 


5:1 


47 


DCPS schools 


14:1 


21:1 


7:1 


133 



* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 “Valid N” refers to the number of schools for which information on a particular characteristic was available. 

2 The comparison group of regular DCPS schools excludes public charter schools, alternative schools, and 
learning centers. 

3 Whenever ranges were given for student-teacher ratio, the midpoint of the range was selected. 

SOURCES: Data on participating private schools drawn from “School Directory, D.C. K-12 Scholarship Program, 
2004-05 School Year,” Washington Scholarship Fund, June 2004, supplemented by the National 
Center for Education Statistics, Common Core of Data. Data on DCPS drawn from the web sites of 
the District of Columbia government and DCPS. 



When it comes to the size of overall enrollments or of individual classes, there are substantial differences 
across schools, although most of these stem from the different grade levels served. Nearly half of the 
participating private schools enroll students in either kindergarten through eighth grade or pre-K through 
eighth grade, with the other half enrolling various combinations of elementary-, middle-, and high-school- 
aged students. It is, therefore, no surprise that enrollments range from 1,000 in a relatively large high 
school to only 12 students in a very small school. The average enrollment of the participating private 
schools is about half the average enrollment of regular DCPS schools. The average ratio of students to 
teachers in the participating private schools is somewhat smaller than the average for DCPS schools. Five 
of the participating private schools (less than 1 0 percent) serve students of only one gender. 

Although there is no data source that allows a systematic examination of the extent to which the 
participating private schools are representative of the entire population of private schools in the District, 
analysis of web sites suggests that there may be some differences. Schools not currently participating in 
the program appear to have slightly higher enrollments than the participating schools, probably because 
more nonparticipating schools serve students in the high school grades. Compared to participating 
schools, nonparticipating schools seem to enroll a substantially smaller share of minorities, have smaller 
student-teacher ratios, and are much less likely to be religiously affiliated. However, these comparisons 
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should be interpreted with caution, because data were available from only 39 of the 51 nonparticipating 
schools and were obtained from difference sources and school years. 



Programs and Services Available at Participating Schools 

The participating private schools offer many but not all of the services and activities that are commonly 
available in public schools (Table 2-3). An overwhelming majority of participating schools offer 
computers, art, a library, tutoring, after-school care, music, and before-school care as part of their 
educational programs. Nearly three-quarters of the participating schools include religious instruction 
and/or worship as part of their school program, and a similar proportion require that applicants complete 
an exam to determine grade placement for enrollment. Nearly 12 percent require applicants to pass an 
entrance exam in order to be deemed admissible to their school. Only 60 percent have a gym and less than 
half have a school cafeteria. Gifted and talented programs are available in about 16 percent of the 
participating private schools. 

Table 2-3. Services, Programs, and Policies Available at Participating 
Schools: 2003-04 



Characteristic or Service 


Number of 
Schools 


Percent 


Valid N 1 


Computers available to students 


49 


96 


51 


Art 


46 


92 


50 


Tutoring 


45 


88 


51 


After care 


45 


87 


52 


Music 


40 


85 


47 


Before care 


40 


77 


52 


Placement exam 


37 


76 


49 


Library 


39 


75 


52 


Religious instruction/worship 


37 


73 


51 


After-school sports 


34 


67 


51 


Gym 


31 


60 


52 


Cafeteria 


24 


47 


51 


Gifted program 


8 


16 


51 


Entrance exam 


6 


12 


52 



“Valid N” refers to the number of schools that provided information on a particular characteristic. 

SOURCE: “School Directory, D.C. K-12 Scholarship Program, 2004-05 School Year,” Washington 
Scholarship Fund, June 2004. 



Participating schools were asked whether they accept and how they accommodate students with 
diagnosed learning disabilities. Private schools are not required to admit students with learning 
disabilities if this would present an undue burden on the schools (according to the standards in the 
Americans with Disabilities Act), unless the private schools receive federal funds (other than those under 
the DC Opportunity Scholarship Program). However, many of the participating schools do admit such 
students. Seventeen of the schools (32 percent) responded that they do not accept students with learning 
disabilities. Twenty-two schools (42 percent) reported that they accept such students and fully include 
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them in the educational environment of the school. An additional eight schools (15 percent) said that they 
accept and provide special services for students with limited or mild learning disabilities, and five 
participating private schools (9 percent) expressed a willingness to accept students with moderate to 
severe learning disabilities. 

It should be noted that the District has a sizable “private placement” program for special education 
students with needs that cannot be served effectively in a DCPS school. The program currently pays the 
full tuition to send 2,595 DC students to specialized private schools or public schools outside of the 
District for educational services specified in student individualized educational plans (IEPs), at an average 
tuition cost of $27,575 per student. 20 At least eight private schools in the District receive private 
placements of special education students from DCPS. 21 None of the eight schools decided to participate 
in the DC Opportunity Scholarship Program this year, perhaps because the scholarship ceiling is below 
the amount that they typically spend to educate students with significant disabilities. 

Table 2-4. Average Tuitions Charged by Participating and Nonparticipating 
Private Schools: 2004-05 





Participating Schools 


Nonparticipating 

Schools 


Average Tuition 1 


Number of 
Schools 


Percent 


Number 
of Schools 


Percent 


Above $7,500 


15 


30 


21 


84 


7,500 or less 


35 


70 


4 2 


16 


Total 


50 


100 


25 


100 



1 For schools that charge a range of tuitions, the midpoint of the range was selected. Tuition rates were 
unavailable for 8 of the participating private schools and 26 of the nonparticipating private schools. 



2 Three schools charged no tuition either because of foundation support or because the school serves 
groups such as DC-placed special education students funded by the government. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCES: Data on participating private schools drawn from “School Directory, D.C. K-12 Scholarship 
Program, 2004-05 School Year,” Washington Scholarship Fund, June 2004. Data on 
nonparticipating private schools were obtained from school web sites. 

Finally, a majority (70 percent) of the private schools that chose to participate in the DC Opportunity 
Scholarship Program the first year charge tuitions that are under the $7,500 maximum amount provided 
through the federal scholarship (Table 2-4). Most ask families to pay annual per student tuitions at or 
below $5,500. Although 30 percent of the participating schools typically charge tuitions above the 
scholarship ceiling, with a couple of exceptions, they agreed to accept a $7,500 scholarship as full 



20 (.<. 

' Proposed FY2005 Budget, Office of the Mayor, Washington, DC,” www.de. gov/mayor/budget/proposed/index2.shtm/ 
(accessed November 11, 2004); “Establishing a Baseline: A Report on the State of Education in the District of Columbia,” 
draft report of the DC State Education Office, June 2004, http://seo.dc.gov/seo/frames.asp7docAseo/lib/seo/SER/ 
State_of_Education_rpt04.pdfandgroup=1507/ (accessed November 11, 2004). 

21 DC School Search web site, www.dcschoolsearch.dc.gov/schools/report_results. asp?report_id=14/ (accessed November 11, 
2004). 
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compensation for enrolling scholarship students, at least for the 2004-05 school year. 22 Nonparticipating 
private schools appear to charge higher tuitions than the participating schools, however, this conclusion is 
based on data for a subset of nonparticipating private schools. 



2.2 Families and Students 

At the same time it was recruiting schools to participate in the DC Opportunity Scholarship Program, 
WSF also was recruiting families and students. Given appropriations of approximately $13 million the 
first year, WSF estimated that approximately 1,800 students could be supported with scholarships. The 
technical advising team hoped for 1,250-1,600 eligible public school applicants in oversubscribed grades 
to generate the conditions for a statistically well-powered experimental impact analysis in future years 
(see Appendix A). 

Although outreach to families in the District began even before WSF was officially designated as the 
program implementer, major efforts were initiated at the end of March when the organization and its 
partners were awarded a 5-year program management grant. These activities included the following: 23 

• Mailing personalized letters to all families that inquired about the program; 

• Mailing notices to officials at Plead Start and after-school centers, the leaders of DC faith-based 
organizations, and officials of various other organizations that serve low-income DC residents 
such as the Greater Washington Urban League; 

• Making personalized phone calls to all families that inquired about the new program or that had 
previously applied to WSF’s or other privately funded scholarship program; and 

• Distributing flyers at community organizations, after-school centers, DC churches, DC Metro 
stations, and several DC neighborhoods populated with low-income families. 

The implementation team then scheduled and advertised a series of application events at the DC 
Convention Center for April 28 to May 1. The application events included a mandatory orientation 
session for families and the distribution of the application form, which collected both baseline 
characteristics of the families and students and information about the families’ eligibility for the 
program. 24 



22 Participating schools that increase their tuition by more than 20 percent in a given year or increase their annual enrollment by 
more than 50 percent are required to submit to an inspection by a task force of education experts before being permitted to 
initiate or continue their participation in the scholarship program. 

23 Shortly after the legislation was enacted in January 2004, Fight For Children (FFC), a 501 (c)3 organization, was designated to 
serve as the interim entity to promote awareness of the program and lay the foundation for its implementation. Among other 
activities, FFC initiated an advertising campaign that featured radio and bus ads, established a web page and call center to 
inform families about the program, and built a database of families that expressed an initial interest in subsequently applying 
for the program and another database of private schools interested in participating in the program. 

As part of the orientation sessions, officials of the WSF explained the requirements of the program and the procedure for 
applying, being selected for, and using a scholarship; members of the technical advising team and Institute of Education 
Sciences (IES) also were present to discuss the evaluation component of the program and explain the rights and responsibilities 
of participants. Also present were officials from the Public School Aid Service (PSAS), a company that verifies eligibility for 
government and school-sponsored scholarship programs. PSAS was under contract to WSF to provide those services and, as 
part of the application events, to train volunteers and assist families in documenting their income and residency eligibility. 
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The application form required parents to confirm that student applicants met all eligibility criteria and to 
provide documentation for verification purposes. For example, income was verified with reference to a 
family’s 2003 income tax return and supporting documents such as official benefits statements pertaining 
to income assistance programs. A student’s entering grade level was verified using a dated report card and 
student birth date (for rising kindergartners). DC residency was established using a family’s tax return or 
a recent utility bill. Families were encouraged to bring full documentation to application events, where 
copy machines were available to them. If they submitted an application that lacked complete 
documentation they were told exactly what additional documentation they needed to provide and where 
they needed to send it. Program staff repeatedly contacted families with incomplete applications in order 
to encourage completion. 

At the conclusion of the series of application events, the program operators decided to extend the 
application period in order to enable more families to apply. The WSF initiated and advertised extended 
evening and weekend office hours to assist applicants on a walk-in basis from May 3 through May 17. 
The implementation team and support organizations staffed a series of 10 application meetings in 
community centers located within the residential boundaries of DC schools that had been declared in need 
of improvement under No Child Left Behind (NCLB). 25 Throughout this period, volunteers assisting the 
implementation team visited homebound parents with disabilities to help them in completing applications. 

On May 26, the implementation team closed new applications to the program. Applicants were given until 
June 1 to provide all of the documentation necessary to complete their applications and verify their 
eligibility for the program. Members of the technical advising team based at Westat consolidated the 
information that had been collected to that point about participating schools and applicants into a master 
database. 

The results of the outreach, application, and eligibility verification process provide some sense of the 
demand for this scholarship program, given its particular requirements and the compressed time period 
within which it was launched in 2004 (Table 2-5). 

• Just over 40,000 students were estimated to be potentially eligible for the program, based on data 
from the U.S. Census regarding the total number of children in grades K-12 in the District with 
family incomes at or below 185 percent of poverty in 2000. 

• Distinct families representing 5,863 school-age children inquired about the scholarship program 
and expressed an interest in applying, according to figures obtained from the nonprofit 
organization that advertised the program before WSF was awarded the program management 
grant. If all of these children were eligible for the program (an unlikely assumption), then they 
would represent approximately 14 percent of the eligible population of students. 

• Applications were submitted on behalf of 2,692 students, representing 7 percent of the eligible 
population and roughly half of the number of student inquiries. 



25 One or more meetings took place from May 4 through May 13 at the Langston Terrace housing project, Tyler House housing 
project, Fletcher Johnson Education Center, Barry Fann Community Center, Barry Fann Recreation Center, and Woodland 
Terrace Community Center. 
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• A total of 1,848 students were confirmed to be eligible for the program; an unknown number of 
additional applicants might have been deemed eligible had they completed their application. 26 
The number of confirmed eligible applicants represents 5 percent of the eligible population, about 
one-third of the inquiries, and 69 percent of all applicants. 

• Of these eligible applicants, 1,330 (72 percent) had been enrolled in DCPS in the 2003-04 school 
year, and 6 percent of those were coming from schools designated as in need of improvement 
under NCLB as of spring 2004. 27 Another 518 (28 percent of all eligible applicants) were already 
attending private schools but satisfied all of the program requirements and, therefore, were 
deemed eligible for scholarships. 



Table 2-5. Number and Percentage of Participants, by Application Status: Spring 2004 













Public 


Private 


Measure 


Eligible 

Base 


Inquiries 


Applicants 


Eligible 

Applicants 


Eligible 

Applicants 


Eligible 

Applicants 


Number of 
students 


40,507 


5,863 


2,692 


1,848 


1,330 


518 


% of base 


100 


14 


7 


5 


NA 


NA 


% of inquiries 




100 


46 


32 


NA 


NA 


% of applicants 






100 


69 


NA 


NA 


% of eligibles 








100 


72 


28 



NOTE: Because the eligible base, inquiries, and applicants included an unknown combination of public and private 

school students, it would not be appropriate to express the number of public or private eligibles as a 
percentage of those bases. 



SOURCE: Figure for the “Eligible Base” is based on data from the U.S. Census, population of the District of Columbia 
age 5 to 17 under 185 percent of the federal poverty line in 2000. The exact number for 2004 is likely to 
differ somewhat from this 2000 figure. Figure for “Inquiries” provided by Fight For Children. Figures for 
“Applicants” and “Eligible Applicants” were drawn from the applicant database, with eligibility determined 
by PSAS. Numbers of applicants from public or private schools were determined by cross-matching the name 
of the school each child was attending against a list of public and private schools in the District compiled 
from the DCPS web site and the NCES Common Core of Data. 



26 A total of 604 student applications were never completed. Applicants provided a number of reasons for failing to complete 
their applications, including that they received a spot in a preferred public or charter school, that they were not confident that 
they would find a desirable private school near their home, or that they realized that they would not be eligible. An additional 
240 student applications were completed, but the applicants were confirmed to be ineligible for the program. 



While there were 79 applicants from the 15 schools designated as SINI in 2003, in advance of the first year lottery in May 
2004, DCPS designated an additional 73 schools as in need of improvement in August 2004, two months after the scholarship 
lottery (See lilln: silicon. k 1 2.dc.us/NCLB. indcx.asp , accessed February 15, 2004). A total of 535 program applicants came 
from 2003-designated or 2004-designated SINI schools, representing 29 percent of all program applicants this first year. Of 
these students, 433 (81 percent) were awarded scholarships. Twice as many Opportunity Scholarships were awarded to 
students applying from 2003- or 2004-designated SINI schools than were awarded to students applying from private schools. 
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3. SCHOLARSHIP AND PLACEMENT LOTTERIES AND INITIAL 
USE OF SCHOLARSHIPS AWARDED 



As discussed in Chapter 1, the DC School Choice Incentive Act requires that a random selection process 
(e.g., a lottery) be used to select scholarship recipients, if the program or specific schools are 
“oversubscribed” — that is, have more demand for them than there are slots available. These lotteries were 
to serve not only as the fairest way to allocate scholarships and placements, but also as the basis for 
creating the randomly determined treatment and control groups for the impact evaluation. In the initial 
year, there were more eligible applicants than new slots only in the upper grades of participating private 
schools. Later, among scholarship recipients, there was more demand than slots at some grade levels in 
particular private schools. 

As a result, the technical advising team assisted the program in designing and executing two separate 
lotteries. The scholarship lottery determined which of the eligible applicants in oversubscribed grades 
would receive scholarships. The placement lottery drew upon the school preferences expressed by the 
parents of scholarship winners to determine which students would receive scarce seats in specific private 
schools. Because of the priority categories specified in the law, SINI scholarship recipients were placed 
first, followed by recipients from non-SINI public schools, then recipients from private schools. This 
chapter describes the design, operation, and results of the lotteries and other placement efforts. 



3.1 Statutory Priorities 

Section 306 of the Act includes congressional priorities to guide the award of scholarships and scarce 
seats to eligible applicants. Program operators are instructed to “provide students and families with the 
widest range of educational options” 28 and to perform the following: 

1. Give priority to students attending a public school in need of improvement under NCLB at the 
time of application to the program; and 

2. “Target resources to students and families that lack the financial resources to take advantage of 
available educational options.” 

Regarding the first priority, the program implementer, the technical advising team, and ED agreed that 
policymakers will want to know if the program is having an effect on all types of students, including 
those who transferred out of NCLB-eligible schools. Thus, it would be desirable to include applicants 
from SINI schools in the rigorous impact evaluation and, therefore, in the random assignment of students 
as part of the scholarship lottery. Awarding scholarships to all SINI students would prevent the evaluation 
from generalizing findings of program impacts to those students from designated low-performing schools 
that are a high priority of the program. 



28 To meet the legislation’s requirement that the program provide a wide range of education options, parents of scholarship 
recipients were encouraged to list up to four preferred private schools for each scholarship student. This was expected to 
improve their likelihood of being able to take advantage of the school choices available to them. 
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However, because the 79 SINI students represented such a small share (4 percent) of the overall applicant 
pool in the first year, there was little possibility of separately analyzing the impacts for this subgroup. 29 
Therefore, for the first year only, every eligible applicant who was currently attending a SINI school was 
automatically awarded a scholarship. 30 The SINI scholarship winners also were the first group of students 
assigned scarce seats in preferred private schools through the subsequent placement lottery. 

In response to the second statutory priority, eligible applicants from non-SINI public schools were 
assigned the second-highest probabilities for scholarship award and were the second group in line for the 
placement lottery, after the SINI applicants. The fact that public school applicants were not yet attending 
private schools but wanted to was inteipreted as evidence that their families lacked the resources to enroll 
in a school of choice. 

The eligible applicants already attending private schools at baseline were given the lowest priority in the 
scholarship and placement lotteries, based on the first two statutory priorities. Thus lotteries had to take 
into account three priority strata: (1) SINI applicants, (2) non-SINI public school applicants, and (3) 
private school applicants. 



3.2 Scholarship Lottery Design and Implementation 

The scholarship lottery was designed to accomplish several objectives. 31 First, the lottery needed to take 
into account the legislated priorities, which essentially give some groups of eligible students a greater 
chance of obtaining a conditional scholarship than other students. Second, the lottery had to lay the 
foundation for the rigorous evaluation of program impacts and effectiveness called for in the law. Finally, 
the lottery had to balance the program’s desire to maximize the use of the scholarships with a need to 
limit the number of disappointed families not able to find school placements; that is, to ensure that there 
would not be too many students seeking the fixed number of private school slots in each grade. Therefore, 
an important part of the lottery design was the creation of scholarship award probabilities that reflected 
the number of applicants in relation to the number of slots available in private schools by grade level 
groups or “bands.” 



1 While there were 79 applicants from the 15 schools designated as SINI in 2003, in advance of the first year lottery in May 
2004, DCPS designated an additional 73 schools as i n need of improvement in August 2004, two months after the scholarship 
lottery (See http: silicon.kl2.dc.us/NCLB/indc\.asn , accessed February 15, 2004). A total of 535 program applicants came 
from 2003-designated or 2004-designated SINI schools, representing 29 percent of all program applicants this first year. Of 
these students, 433 (81 percent) were awarded scholarships. Twice as many Opportunity Scholarships were awarded to 
students applying from 2003- or 2004-designated SINI schools than were awarded to students applying from private schools. 



10 With more schools identified as SINI and the program implementer’s plan for aggressive recruitment of students from these 
schools in 2005, it is likely that a much higher number of students in this priority group will apply to the DC Opportunity 
Scholarship Program next year. Under these conditions, the group will be given highest but not absolute priority and will be 
part of the random assignment process that will enable that important category of students to be included in the rigorous 
impact evaluation. 



31 This first stage lottery results in a “conditional scholarship” for lottery winners, because to use the scholarship, families must 
ultimately go through a placement process at any participating school in which they are interested. 
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Applicants and Available Private School Slots, by Grade Band 



The awarding of scholarships to public 
school applicants for the first year was 
constrained by the limited availability of 
new private school seats at certain grade 
levels, not necessarily by the amount of 
funds available for scholarships. The 
distribution of eligible public school 
applicants by grade did not closely match 
the distribution of new slots by grade in 
participating private schools (Figure 3-1): 

• In grades K to 5, more private 
school seats were available than 
eligible public school applicants 
interested in filling them (i.e., 
there was more than one slot 
available to each such applicant in 
the elementary grades). 

• Grades 6 to 8 were modestly oversubscribed given seat availability. 

• Grades 9 to 12 were severely oversubscribed given seat availability. 

Since eligible private school applicants already held slots in their private schools, they were not 
constrained by slot availability in the same way as public school applicants. 



Figure 3-1 

Eligible Public School Applicants and Available 
Private School Slots, by Grade-Level Band: 
Spring 2004 and Fall 2004 
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Creating Probabilities of Scholarship Award 

The first step in generating the probabilities for the scholarship lottery was to create base probabilities that 
matched the ratio of slots to applicants. The base probability for K to 5 public school applicants was 
above 1 00 percent, meaning that all would receive scholarships. The base probabilities for eligible public 
school applicants entering grades 6 to 8 and 9 to 12 were 66 percent and 31 percent, respectively 
(Table 3-1). 

In finalizing the probabilities, the technical advising team also took into account that some students 
awarded scholarships are likely to choose not to use them. Low-income inner-city families are highly 
mobile, and some of them would likely move out of the District before the start of the school year, 
making them ineligible for the program. Other scholarship winners might choose to remain in their 
current public school or enroll in another public school in the District such as a charter school. Therefore, 
a decision was made to assign more scholarships than the number of slots available in the middle and 
high school grades. 32 A previous evaluation of the partial-tuition scholarship program for elementary 



12 No overassignment of scholarships was necessary at the elementary grades, since more slots were available than there were 
eligible applicants. 
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Table 3-1. Eligible Public School Applicants and Available Private School 
Slots: Spring 2004 and Fall 2004 



Grade-Level Band 1 


Number of 
Eligible 
Applicants 


Number of 
Available Slots 


Ratio of 
Slots/Applicants 


K-5 


796 


968 


1.22 


6-8 


356 


236 


0.66 


9-12 


191 


60 


0.31 


Totals 


1,343 


1,264 


0.94 



Forecasted grade level of each applicant based on parental reports of “Grade entering for the 2004-05 school 
year” on the application form. All applications for students forecasted to be entering kindergarten in 2004-05 
were examined to determine if the child’s age was above the cutoff for kindergarten enrollment. 

SOURCES: Figure for the “Number of Eligible Applicants” drawn from the applicant database. “Number of 
Available Slots” based on official reports from all participating private schools as of June 1, the slot 
cutoff for purposes of designing the scholarship lottery. 

school students in New York City reported that 18 percent of students awarded scholarships there did not 
use them in the first year. 33 We might expect that the rate of non-usage would be somewhat lower for the 
middle school students who receive Opportunity Scholarships, since they cover the full tuition, but 
somewhat higher for the high school students, who were able to select from relatively few participating 
schools in the first year. 34 

Based on these expectations, we set the scholarship “overage” as 1 6 percent for middle school students 
and 28 percent for high school students. Thus, the overall scholarship probability for middle school 
students applying from public schools was 82 percent (66 percent + 16 percent) and the probability for 
public high school applicants was 59 percent (31 percent + 28 percent). Since the SINI public school 
applicants within these grade bands were automatically awarded scholarships this first year, the award 
probabilities for the non-SINl public school applicants were somewhat lower than the grade-band 
average, depending on the proportion of slots that automatically were awarded to SINI public applicants. 



33 Howell et at., The Education Gap, 44. 

34 Only nine participating private schools served students in any of the high school grades. 
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Results of the Scholarship Lottery 



Once these decisions had been made, the 
scholarship lottery operated as a series 
of mini-lotteries for each of the priority 
strata and grade bands within them. The 
computer program applied separate 
selection probabilities to the groups of 
eligible applicants who met the various 
priority and grade-band criteria (Figure 
3-2 and Table 3-2). 



• Because the award of 
scholarships to the public school 
applicants in grades K to 5 was 
not constrained by the number 
of slots available, all those 
applicants were automatically 
awarded scholarships. 

• The SINI public school applicants, representing the highest statutory priority, all received 
scholarships even if they were entering the slot-constrained middle and high school grades. 

• A modest number of non-SINI public school applicants entering middle school and a moderate 
number of non-SINI public school applicants entering high school were randomly assigned to the 
control group (81 and 1 12, respectively). 

• A total of 216 of the applicants already attending private schools were awarded scholarships. This 
last number was determined by ED and the DC Mayor, based on setting award probabilities at 
about half of the probabilities for non-SINI public school applicants in the relevant grade bands. 



Figure 3-2 

Scholarship Probability, by Type of Applicant and 
Grade-Level Band: Spring 2004 
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Table 3-2. Lottery Probabilities and Assignments, by Applicant Type and Grade-Level 
Band: Spring 2004 



Type of Applicant 


Grade-Level Band 


Total 1 


K - 5 


6-8 


9-12 


Public SINI 










Eligibles 


24 


20 


35 


79 


Probability 


100.0% 


100.0% 


100.0% 


100.0% 


Scholarships 


24 


20 


35 


79 


Nonrecipients 


0 


0 


0 


0 


Non-SINI Public 










Eligibles 


772 


336 


156 


1,264 


Probability 


100.0% 


75.9% 


28.2% 


84.7% 


Scholarships 


772 


255 


44 


1,071 


Nonrecipients 


0 


81 


112 


193 


Private 










Eligibles 


239 


156 


110 


505 


Probability 


55.2% 


41.7% 


17.3% 


42.8% 


Scholarships 


132 


65 


19 


216 


Nonrecipients 


107 


91 


91 


289 


Totals 










Eligibles 


1,035 


512 


301 


1,848 


Probability 


89.7% 


66.4% 


32.6% 


73.9% 


Public Scholarships 


796 


275 


79 


1,150 


Private Scholarships 


132 


65 


19 


216 












Total Scholarships 


928 


340 


98 


1,366 


Public Nonrecipients 


0 


81 


112 


193 


Private Nonrecipients 


107 


91 


91 


289 


Total Nonrecipients 


107 


172 


203 


482 



The totals here for combined public school applicants (79+1264 = 1343) and private school applicants (505) differ from the 
figures of 1330 and 518 presented elsewhere (e.g., in Table 2-5). The reason is that 13 applicants were originally classified as 
attending non-SINI public schools based on parental responses to the school type and name of school questions in the baseline 
survey. In all 13 cases, the parent indicated that the child was attending a public school and provided either no school name (4 
cases) or a school name that did not obviously signify the type of school (9 cases — 4 of which were preschools). Further 
investigations by the technical advising team indicated that these 13 students were attending private schools or private 
preschools, and their school type classification was corrected accordingly. 

NOTE: The shaded boxes represent the randomization of the impact analysis sample. 

SOURCE: Evaluation database (based on information from multiple sources: (1) the DC Opportunity Scholarship 
Program applications, (2) the program operator’s files, and (3) data files from DCPS). 
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As a result of the automatic awarding of scholarships to some students and the randomization process in 
oversubscribed grades: 15 

• A total of 1,366 eligible applicants were awarded program scholarships in the initial year. 16 They 
comprise the program sample that will be the subject of performance reporting in future 
evaluation reports. Of these, 1,150 (84 percent) were applicants from public schools, and 216 
(16 percent) were applicants from private schools. 

• A total of 299 (22 percent) of the students in the program sample comprise the treatment group 
for the impact sample (Table 3-3). They were all public school applicants entering grades 6 to 
12, where the number of eligible applicants exceeded the number of available private school slots. 

• A total of 482 students were randomly assigned to nonrecipient status. Of the students who did 
not receive scholarships through the lottery, 193 (40 percent) were public school applicants who 
will comprise the control group for the impact analysis. We will track their academic 
achievement and experiences over time and compare them with those of the 299 scholarship 
recipients in the treatment group to assess program impact. 

• The remaining 289 (60 percent) nonrecipients who were private school applicants are members of 
neither the program sample (because they did not receive a scholarship) nor the impact sample 
(because they are nonrecipients likely to continue to attend private school). Thus, this group of 
eligible applicants from the 2003-04 cohort will not be followed in the future as part of the 
evaluation. 

In sum, the scholarship lottery produced two groups of students for purposes of meeting statutory 
requirements for the DC Choice Opportunity Scholarship program evaluation. The 1,366 scholarship 
recipients will be the subject of annual performance reporting. The 492 public school applicants who 
were entering grades subject to random assignment will contribute to the annual impact analysis (see 
shaded cells of Table 3-3): these include 299 students assigned to the treatment group (and also included 
in the performance reporting sample) and 193 students assigned to the control group. The 289 private 
school applicants who were not awarded scholarships belong to neither group. Since they previously 
attended and presumably will continue to attend private school using resources outside of the scholarship 
program, following these students after baseline would not contribute meaningfully to the evaluation. To 
conserve resources, this group of initial applicants will not be part of the evaluation going forward. 



35 The customized computer program designed by the technical advising team to implement the lottery was executed at the WSF 
offices on June 16, 2004. Members of the public and officials from ED were present to witness the lottery. A diagnostic 
program confirmed that the lottery operated exactly as designed, including the creation of comparable treatment and control 
groups for applicants in the oversubscribed grades 6 to 12. 

36 Of the 1,366 scholarship winners, 1,249 were selected via the initial execution of the lottery program. As described in Chapter 
2, after the lottery, additional applicants became eligible for the program either by providing necessary documentation that 
completed their original application or through an appeals process conducted by the WSF. These additional eligible applicants 
were automatically awarded scholarships if they were attending SINI public or public school grades that were not 
oversubscribed (i.e., entering K-5), otherwise they were randomly assigned to the scholarship or control group using the same 
probabilities that had been applied to the initial group of eligible applicants. This subsequent lottery process resulted in an 
additional award of 117 scholarships and eight additional control group members, since most of the newly eligible applicants 
qualified for the automatic scholarship awards. 
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Table 3-3. DC Opportunity Scholarship Program Applicants, by Receipt of Scholarship and Inclusion in the Impact Analysis 
Sample: Spring 2004 



School Type at 
Application and 
Grade Band 


Eligible 

Applicants 


Applicants who Received Scholarships 
(Performance Reporting Sample) 


Applicants who Did Not Receive Scholarships 


Non-Impact 

Sample 

Recipients 


Impact 

Sample 

Recipients 


Total 

Scholarship 

Recipients 


Impact 

Sample 

Nonrecipients 


Non-Impact 

Sample 

Nonrecipients 


Total 

Nonrecipients 


SIN1 
















Grades K-5 


24 


24 


0 


24 


0 


0 


0 


Grades 6-8 


20 


20 


0 


20 


0 


0 


0 


Grades 9-12 


35 


35 


0 


35 


0 


0 


0 


Subtotal 


79 


79 


0 


79 


0 


0 


0 


Non-SINI Public 
















Grades K-5 


772 


772 


0 


772 


0 


0 


0 


Grades 6-8 


336 


0 


255 


255 


81 


0 


81 


Grades 9-12 


156 


0 


44 


44 


112 


0 


112 


Subtotal 


1,264 


772 


299 


1,071 


193 


0 


193 


Private 
















Grades K-5 


239 


132 


0 


132 


0 


107 


107 


Grades 6-8 


156 


65 


0 


65 


0 


91 


91 


Grades 9-12 


110 


19 


0 


19 


0 


91 


91 


Subtotal 


505 


216 


0 


216 


0 


289 


289 


Totals 


1,848 


1,067 


299 


1,366 


193 


289 


482 



NOTE: The shaded boxes represent the randomization of the impact analysis sample. 



SOURCE: Evaluation database (based on information from multiple sources: (1) the DC Opportunity Scholarship Program applications, (2) the program operator’s files, 
and (3) data files from DCPS). 




However, the impact sample in the first year of the program is not, on its own, sufficiently sizable to 
identify program impacts unless they are very large. The statistical power analysis conducted for the 
evaluation (see Chapter 1 and Appendix A) suggests this initial treatment (scholarship) and control (no 
scholarship) group in the impact sample is too small for the evaluation to reliably draw conclusions about 
any differences in achievement outcomes that might be expected from an intervention of this kind. 17 
Instead, the treatment and control groups from the first year lottery will be combined with those from the 
lottery for second year applicants, expected in April 2005, to bring more evidence to bear on the rigorous 
evaluation of program impacts. 



3.3 Placement Lotteries 

To receive a seat under the DC Opportunity Scholarship Program, scholarship winners were instructed to 
identify and visit each private school that they were interested in attending. To facilitate this process, the 
WSF and its partners conducted school fairs where parents and children were given the opportunity to 
meet with representatives from participating schools to learn about the programs and admission 
requirements to enter the schools. 18 Scholarship winners were strongly encouraged to meet with 
representatives from more than one school to maximize their chance of placement in a school of their 
choice. 

It was a requirement of the scholarship program that to receive a seat in a private school, scholarship 
winners must (1) visit with private school representatives and secure a provisional seat in that school and 
(2) complete and submit a school preference form to the WSF listing schools by first, second, third, and 
fourth choice. 

Although the program operators initially planned to conduct a single placement lottery, this was not 
possible because many participating schools were late to finalize the number of seats available to 
scholarship winners, and many families failed to complete their school visits and other admissions 
requirements by the lottery deadline of July 12. 39 A decision was made to go forward with the lottery so 
that the substantial number of families that had completed the school search process in time could be 
quickly informed about where their children would be attending school in the fall. The program 
implementer determined that they would assist the remaining scholarship recipients in completing their 
search and in finding appropriate placements. As a result, placements were made using a combination of a 
lottery for children meeting all of the requirements by the specified deadline followed by WSF placing 
children in schools on a flow basis as schools finalized their seat openings and scholarship winners 
completed all of the requirements of their school search. For the purposes of the placement process, no 
distinction was made between scholarship winners in the impact sample and those in the non-impact 
sample. 



37 As discussed in Chapter 1 and Appendix A, statistical power analysis is an important part of evaluation design and 
demonstrates how well the design will be able to distinguish real impacts from chance differences. The analysis takes into 
consideration such factors as the size of the impact expected, the proportion of students anticipated to be assigned to the 
treatment and control groups, and the likely attrition from the study’s data collection. 

38 The law allows participating private schools to use the same admissions criteria for federal scholarship recipients as for other 
applicants to the school. 

39 

Because of delays in obtaining a final list of available seats at participating schools, the actual computerized placement lottery 
could not be run until August 6. 
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Lottery Methodology Used To Distribute Seats to Scholarship Recipients 

To ensure that scholarship winners were assigned seats in a school without bias or favoritism, Westat 
programming staff wrote and thoroughly tested a computer application program that distributed seats in 
accordance with the following rules: 

• Scholarship winners currently attending a public school but already admitted to a specific private 
school before the DC Opportunity Scholarship Program began (i.e., “pre-admits”) were 
automatically given a seat in the school in which they were pre-admitted. The automatic award of 
these seats had no effect on the seats available to other scholarship winners. 

• Scholarship winners already enrolled in a participating private school who submitted a form to 
WSF indicating that they wished to remain in that school were also automatically assigned a seat. 
As noted earlier, there was no effect on seats available to the remaining scholarship winners. 

After eliminating these students from the seat competition, the remaining students were assigned seats 
under the following rules: 

• Students were organized into three categories corresponding to the priority groups used for the 
scholarship lottery (i.e., all students attending SINI schools were grouped together, all students in 
non-SINl public schools were grouped together, and all students in private schools who were 
seeking to change schools were grouped together). 

• The students in each separate category were randomly ordered. 

• Once the random order was established within each group, the three groups were placed in a 
single file for school assignment with the SINI public students first, followed by non-SINI public 
students, and finally private school students seeking to switch schools. 

• Placement was based on the school preferences— up to four schools could be named and ranked in 
order of priority— indicated by each family for each child. Only schools that had agreed to accept 
the child could be assigned as part of the placement lottery. Since a seat may not have been 
available in a student’s first school preference, an attempt was made to place the student in their 
second preference, then third, and so on until all preferences were exhausted. 

• The seat assignment described here was used for each child, with the exception of families with 
more than one scholarship student who wanted to place their children in the same school. 40 When 
this occurred, the children were processed together. The first school preference was examined to 
determine whether seats were available at that school for all scholarship children in the family. If 
not, the second preference was examined followed by the third, and so on until all school 
preferences were checked. If a family placement was not possible, then children were placed 
individually, as described earlier. 



40 There were 348 families that won scholarships for multiple children. Of these multi-award families, 129 submitted school 
preference forms in time for the placement lottery. A total of 1 19 of the 129 preference forms (92 percent) included a request 
that all the children in the family be placed in the same school. 
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Results of the First Placement Lottery 

The total number of scholarship winners at the time of the first placement lottery was 1,277. 41 However, 
5 13 of them were not prepared to participate in the first placement lottery because they failed to visit with 
school representatives to secure a seat, failed to return a school preference form by the deadline, or both. 
Some of the reasons given include that they could not attend school fairs because of scheduling conflicts, 
transportation problems, insufficient length of time to visit with school representatives, could not afford 
to miss work to visit schools, and some simply lost interest in the program. 

Next, 196 scholarship winners automatically received a seat in a particular private school. These included 
(1) students who were currently attending a public school but who had already been admitted to a 
participating private school for the fall term prior to the late-April program application period, and (2) 
applicants from participating private schools who completed a school preference form indicating that they 
intended to remain in their current school. In neither of these cases did a new slot need to be taken at a 
participating private school. That left a total of 568 students who were entered into the placement lottery 
to compete for available seats in private schools. 42 

The computerized placement lottery assigned seats to 549 of the 568 students seeking seats in 
participating private schools for the school year beginning in fall 2004 (Table 3-4). As predicted by the 
legislation, there were more scholarship recipients applying to some private schools than could be 
accommodated. However, of the 568 students who were competing for new slots, 531 (93 percent) were 
assigned their first choice school. Another 3 percent wound up in their second or third preferred school. 43 
Because the school preferences clustered around certain geographically or programmatically desirable 
schools, 19 students (3 percent) could not be placed in a preferred school by the computerized placement 
lottery. 



Followup to the Placement Lottery 

Since the start of the school year was rapidly approaching and some participating schools were making 
additional slots available to scholarship students, the remaining 295 scholarship students who completed 
their school search after the operation of the placement lottery were assigned to available slots in private 
schools by WSF on a case-by-case basis. The added slots also permitted WSF eventually to place almost 
all students who could not be placed by the initial placement lottery, including every student who 
remained interested in participating in the program. Parents and children were contacted either by 
telephone or letter notifying them of their successful placement. 



41 Another 89 students were awarded scholarships after the first placement lottery because they were deemed eligible upon 
finally completing their application or as a result of the appeals process. All of these students were public school applicants 
entering grades K-5 and thus were automatically awarded scholarships upon being certified eligible for the program. 

42 The number of scholarship winners listed for SINI public, Non-SINI public, and Private schools differ from the numbers 
shown in the scholarship lottery. For the placement lottery, if a parent indicated on the school preference fonn that he/she 
wanted all of his/her children placed in the same school and the children were currently attending schools that fell under two or 
more of the classifications below, all children in the family were listed in the highest priority and most advantageous 
classification. For example, if there were two children in a family and one attended a SINI school while the other did not, both 
children were classified as SINI. This was done so that SINI students would not be penalized for applying along with one or 
more non-SINI siblings. 

4j All the students who listed a fourth school choice were placed in their first, second, or third choice school. 
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Table 3-4. 



Results of the Computerized Placement Lottery, by Priority Group: 
Spring 2004 



Priority Group 


Students 

Competing 

for 

Available 

Seats 


Students 

Assigned 

First 

Choice 1 


Students 
Assigned 
Second or 
Third 
Choice 


Total 

Students 

Placed 


SINI 


47 


44 


0 


44 


Non-SINI Public 


503 


471 


18 


489 


Private 


18 2 


16 


0 


16 


Total 


568 


531 


18 


549 



1 If a child submitted an invalid first school preference, and the second school preference was valid and they were able 
to obtain a seat in that school, the child is counted as being placed in their first preference on this table. 

2 These students were private school scholarship winners who wanted to switch to a different participating private 
school and, therefore, were entered into the competition for new private school seats. 

SOURCE: Applicant database and school preference forms submitted by parents. 



At the end of the placement process, 1,040 scholarship students were assigned available seats in 
participating private schools selected by their parents (Table 3-5). Every scholarship winner, whether in 
the impact analysis treatment group or not, who completed a school search and remained interested in the 
program was eventually placed in a private school chosen by the family. 



3.4 Scholarship Usage 

Of the 1,366 students awarded scholarships through the computerized scholarship lottery and appeals/ 
extended completion period, 1,040 obtained a placement, and 1,027 had matriculated at a preferred 
private school as of September 10, 2004. This represents an initial scholarship usage rate of 75 percent for 
the program sample that will be used for performance reporting. This program-wide scholarship usage 
rate is at the high end of the range for initial usage among the previous large-scale school voucher or 
private scholarship programs. 44 

The initial scholarship usage rate for the 299 members of the treatment group in the impact sample was 
somewhat lower, at 62 percent. The lower rate for this group, in comparison to both the overall program 
sample and that of other scholarship programs reflects two factors: 



44 The initial scholarship usage rate was 82 percent in New York City, 78 percent in Dayton, Ohio, and 68 percent in 
Washington, DC, for the earlier experimental evaluation of private scholarship programs in those three cities. Voucher usage 
rates were 61 percent in the first year of the Milwaukee Parental Choice Program and estimated to be about 50 percent in the 
first year of the Cleveland Scholarship and Tutoring Program. See Howell et al., The Education Gap, 44; John F. Witte, First 
Year Report: Milwaukee Parental Choice Program (University of Wisconsin-Madison, November 1991), 3; John F. Witte, 
Andrea B. Bailey, and Christopher A. Thom, Second Year Report: Milwaukee Parental Choice Program (University of 
Wisconsin-Madison, December 1992), 8; Paul E. Peterson and Bryan C. Hassel, eds., Learning From School Choice 
(Washington, DC: Brookings, 1998), 360. 
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Table 3-5. 



Final Results of Lottery and WSF Student Placements, by Priority Group: 
Spring 2004 



Priority 

Group 


Scholar- 

ship 

Winners 


Failed To 
Complete 
School 
Search 


Placed 

Through 

Lottery 


Not 

Changing 

Schools 1 or 
Placed by 
WSF 


Total 

Students 

with 

Placement 


Completed 
Search but 
Unplaced 


SINI public 


79 


28 


33 


18 


51 


0 


Non-SINI 

public 


1,071 


276 


497 


298 


795 


0 


Private 


216 


22 


19 


175 


194 


0 


Total 


1,366 


326 


549 


491 


1,040 


0 



1 Not changing schools includes public school applicants who obtained admission to a particular private school prior to applying 
to the program and private school applicants continuing in their previous school. There were 196 students in this category. 

SOURCE: Computerized placement lottery database (Westat) and WSF placement records. 



• Scholarship usage rates vary significantly by grade level, with the highest usage among K-5 
students (Appendix D, Table D2). Since the treatment group in the impact sample does not 
include any K-5 students, the initial usage rate for the treatment group is lower than the rate for 
the program sample as a whole and lower than that of other scholarship programs, almost all of 
which focused on elementary grades. 

• In addition, about 1 6 percent of the DC Opportunity Scholarship recipients are students who were 
already attending private schools when they applied and thus were virtually certain to use their 
scholarships. These students are not represented among either the 299 members of the treatment 
group or recipients of other scholarship programs. 

Students who chose to take advantage of their scholarships were distributed throughout the participating 
schools, although there was some clustering by religious affiliation and location. For example, although 
about half of the schools are affiliated with the Catholic Church, 6 1 percent of scholarship users enrolled 
in a Catholic school. Seventeen percent of the students using scholarships chose a non-Catholic religious 
school. The proportion of scholarship students attending nonsectarian private schools, 22 percent, is high 
compared to previous government-sponsored school choice programs. 45 



43 For example, only 4 percent of the students participating in the Cleveland Scholarship and Tutoring Program in 1999-2000 
attended secular private schools, a fact that figured prominently in the deliberations surrounding the constitutionality of that 
school choice program (See Zelman, Superintendent of Public Instruction of Ohio, et al. v. Simmons -Harris et al. 536 U.S. 647 

(2002), p. 1). 
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This pattern of scholarship use in the 
District is a function of the types of 
private schools established and willing 
to participate in the program and 
differences in the numbers of new seats 
that schools of different types made 
available. 

As with the distribution of participating 
schools described in Chapter 2, the 
schools that scholarship students have 
selected are slightly concentrated in a 
few wards but are also found 
throughout the District 46 (Figure 3-3). 

Almost one-quarter of scholarship users 
are attending private schools in Ward 4, 
and Ward 6 is hosting nearly 20 
percent. The remaining scholarship students are scattered throughout the city, with Ward 2 private schools 
serving the smallest proportion of participating students. 



Figure 3-3 

Participating Private Schools Selected by 
Scholarship Recipients, by Ward: Spring 2004 




46 The ward locations of schools were detennined using the NCES Common Core of Data and the Citizen Atlas at the District of 
Columbia Government web site, available at citizenatlas.dc.gov/atlasapps/reporthometab.aspx. 
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4. CHARACTERISTICS OF PROGRAM APPLICANTS 



The implementation and impacts of most programs are shaped by the characteristics of those who apply to 
participate. The DC Opportunity Scholarship Program is no exception; the law’s provisions make it clear, 
for example, that the program is intended to serve economically disadvantaged families. In determining 
and interpreting program effects later on, it is important to know how well the program is targeted to 
disadvantaged families and, beyond that, what types of families and students apply, win scholarships in 
the lottery, and choose to use them to enroll in a private school. This chapter describes the 1,848 eligible 
applicants to the DC Opportunity Scholarship Program in part by comparing them, and subsets of them, 
to several policy-relevant groups. The chapter concludes with some initial evidence regarding the extent 
that public and private schools in the District are experiencing a significant loss or gain of students due to 
the first year of program implementation. 



4.1 Public School Applicants Compared to Public Nonapplicant Students in the District 

There are several reasons to examine the eligible public school applicants to the DC Opportunity 
Scholarship Program in relation to other DCPS students. Most importantly, the comparison provides a 
context for considering the kinds of students who might be attracted to the program in future years in the 
District or to a similar program in other locations. 

Of course, program applicants differ from public nonapplicant students in important ways, some by 
design of the program. Applicants must have family incomes at or below 185 percent of the federal 
poverty line. In addition, the majority of applicants were young students, in the early elementary grades, 
while DCPS students are somewhat more evenly spread out across grade levels. These differences are 
important because they can affect average test scores, racial/ethnic group membership, or other 
characteristics of interest. As a result, we compare applicants to several different subgroups of DCPS 
students, in addition to DCPS students overall, and make comparisons within certain grade bands 
whenever it is practical to do so. 

Because the evaluation’s primary outcome measure is academic achievement, it is useful to understand 
whether applicants to the DC Opportunity Scholarship Program started out with higher, lower, or similar 
levels of achievement than nonapplicants. This focus necessarily limits the analysis to the 784 eligible 
applicants who participated in the accountability testing in DC public and charter schools in the spring of 
2004 and could be identified conclusively in the database provided by DCPS. 47 The analysis cannot 



47 The DCPS Office of Communications and Public Information provided the data file to the technical advising team consistent 
with the requirement in the legislation that all schools and school systems that are covered by the DC School Choice Incentive 
Act of 2003 cooperate with the program evaluator in providing information necessary to the evaluation. The DCPS 
accountability database included demographic information on 1077 eligible applicants, representing 80 percent of all public 
school eligible applicants. The data base also included a reading or math test score for 784 of these program applicants, or 58 
percent of all eligible public school applicants to the program. Of the 42 percent of applicants without such data: (1) 65 percent 
are students entering grades K-l, for whom there are no DCPS tests offered; (2) another 28 percent are students entering 
grades 2 and 3 who are not required to take the DCPS accountability tests, although some did and are included in the 
comparisons; (3) the remaining 17 percent are students entering grades 4 through 12. To examine whether there was any 
substantial bias in our analysis, we compared the demographic characteristics of students who were tested by DCPS with those 
who were not tested, both within and across the applicant and nonapplicants samples, using a t test for difference of means to 
identity statistically significant differences. The testers did not differ significantly from the nontesters on any characteristic, 
except for grade, since testing is mandatory only in grades 3 and higher. Therefore, we are confident that tester/nontester bias 
does not affect the comparisons that we make here. 
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include private school applicants, for whom baseline (pre-application) test scores are not available or are 
not comparable to testing in DCPS. 48 

The accountability database contains the results of DCPS administration of the Stanford Achievement 
Test, Version 9 (SAT-9) to all students in grades 3 through 11, and some students in grades 1 and 2, in the 
spring of 2004. These tests covered the areas of reading and mathematics, with results expressed in terms 
of National Percentile Ra nks (NPRs). The NPR for a particular student test score describes the 
percentage of students who scored below that level on a nationally normed test for the specific grade 
level, subject, and testing period that applies to the student score. 49 Expressing test-score norms in this 
way makes them comparable across grade levels. The DCPS database also includes a modest number of 
student demographic characteristics with which to organize comparisons. 



All Public School Applicants Compared to All Nonapplicant DCPS Students 

To answer the question — what kinds of students apply to the DC Opportunity Scholarship Program — 
program applicants as a whole were considered in relation to all DCPS students who did not apply to the 
program. This analysis yields the following conclusions (Table 4-1): 

• Applicants performed similarly to DCPS nonapplicants in both reading and mathematics. 50 

• Applicants are somewhat more likely (16 percent) than public nonapplicant students (14 percent) 
to have been enrolled in special education. 51 

• Public school students attracted to the DC Opportunity Scholarship Program were more likely to 
be African American than were nonapplicant public school students (92 percent vs. 85 percent) 
but less likely to be Hispanic (6 percent vs. 9 percent) or other races (2 percent vs. 6 percent). 

• Applicants are substantially more economically disadvantaged than are DCPS students overall; 
applicants were more likely to be enrolled in the federal Free or Reduced-price Lunch Program 
(FRL) than were nonapplicants (85 percent vs. 68 percent), a result that is not surprising given the 
income ceiling for program eligibility. 52 



48 There were not sufficient evaluation resources or time during the first year of the program to consider having the evaluation 
team test all program applicants, including those from private schools. 

49 For example, the score average of nearly 47 NPRs for all K to 5 applicants in reading is about 3 percentile points below the 
national median for those grades in reading, as approximately 47 percent of students across the country who took the reading 
test last spring in those grades scored lower than the average for the program applicants. 

50 Test scores were missing from about one-quarter of the sample, so these results should be interpreted with caution. However, 
as discussed in footnote 47, the background characteristics of testers and non-testers do not indicate any clear biases that would 
necessarily undermine the comparisons. 

51 Students generally are classified as enrolled in special education if they have an individualized education plan consistent with 
the federal Individuals with Disabilities Education Act. As with the test score comparison, the significant amount of missing 
data regarding disability status requires that results be interpreted with caution. 

42 Although all applicants who were deemed eligible for the program were verified as having family incomes that made them 
eligible for the federal lunch program, some of them were not enrolled in that program. It is well established that not all 
students who are eligible for the lunch program enroll in it, and the percentage of eligible students who decline to enroll tends 
to increase in the upper grades. For example, our data indicate that 14 percent of K-5 eligible applicants, 16 percent of grades 6 
to 8 eligible applicants, and 19 percent of grades 9 to 12 eligible applicants were not enrolled in the lunch program (Appendix 
B, Table Bl). Any bias that this underenrollment introduces into the sample is likely to be roughly equal for both the program 
applicant and nonapplicant groups, so it probably does not bias the comparison between the two. 
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Table 4-1. Characteristics of DC Public School Students, Program Applicants 
Versus Nonapplicants: Spring 2004 



Characteristic 


Applicants 


DCPS 

Sample 


Difference 


Baseline Test Scores 1 








Average Reading Percentile 


41.5 


40.1 


1.4 


Percent missing 


26 


26 




Average Mathematics Percentile 


47.2 


46.3 


1.0 


Percent missing 


25 


25 




Percent in Special Education 


16 


14 


2 * 


Percent missing 


24 


23 




Percent, by Race 








African American 


92 


85 


g** 


Hispanic 


6 


9 


_4** 


Other race 2 


2 


6 


_4** 


Percent missing 


1 


1 




Percent, by Gender 








Female 


51 


50 


0 


Percent missing 


0 


1 




Percent Participating in Free/Reduced- 








price Lunch Program 


85 


68 


27 ** 


Percent missing 


2 


2 




Sample size 


1,077 


66,868 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in tenns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes students who were identified as white, Asian, American Indian, or Alaska Native. 

NOTE: Detail may not sum to totals because of rounding. Applicant sample includes all applicants identified in the DCPS 

database, regardless of whether or not they participated in the accountability testing. 

SOURCE: Accountability testing database for District of Columbia public and charter schools, DCPS 
Office of Communications and Public Information. 
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All Public School Applicants Compared to Economically Disadvantaged Nonapplicant DCPS Students 



Since, by virtue of the eligibility criteria, program applicants are more economically disadvantaged than 
DCPS students overall, it is appropriate to consider the applicants’ characteristics relative to those of 
public school students who are similarly disadvantaged. The income criterion for the DC Opportunity 
Scholarship Program — at or below 185 percent of the federal poverty line — is similar to that defining 
eligibility for enrollment in the federal FRL program or for students that are Title 1 eligible (Table 4-2). 53 

Using those indicators of economic disadvantage for both applicants and nonapplicants we conclude the 
following: 

• Program applicants have higher test score performance than economically disadvantaged 
nonapplicants. Applicants scored nearly four NPR points higher in both reading and mathematics 
than similar nonapplicants. 54 

• Among economically disadvantaged students, there is no statistically significant difference 
between applicants and nonapplicants in the proportion enrolled in special education. 

• Lower income applicants remain somewhat more likely to be African American (93 percent) and 
less likely to be Hispanic (6 percent) than nonapplicant public school students with similar 
income levels (88 percent and 10 percent, respectively). 

• By design, the two groups are similar regarding their eligibility for the FRL program. 



All Public School Applicants Compared to Economically Disadvantaged Nonapplicant DCPS Students, 
by Grade Bands 

In addition to their economic circumstances, program applicants differ from DCPS students overall in 
terms of their grade levels: applicants, including public school applicants, are more likely to be in the 
elementary grades and less likely to be in the high school grades than are DCPS nonapplicants. To make 
the groups as comparable as possible, we, therefore, present data on eligible public school applicants in 
each of the three grade bands used for the lottery and nonapplicant DCPS students in the identical grade 
bands who were income-eligible for the program. Students are assigned to a subgroup based on the grade 
they were forecasted to be entering in the fall of 2004, as is the case with all of the grade-level 
comparisons in this report. We include here the sample that is limited to economically disadvantaged 
students, since controlling for family income, which is associated with test scores, generates a more 
reliable comparison. A comparison by grade band using the full sample of students, including students 
above the FRL eligibility ceiling, is presented in Appendix B (Table Bl). 



53 The data from the accountability test score file for charter students did not include information on FRL status. The charter 
student data did include an indicator for whether the student was eligible for Title 1 . We confirmed with the DC Office of 
Communications and Public Information that the income cutoffs for Title 1 and FRL in the District are identical. Thus, we 
included charter students in the comparison limited to FRL if they were designated as Title 1 eligible. 

54 Test scores were missing from about one-quarter of the sample, so these results should be interpreted with caution. However, 
as discussed in footnote 47, the background characteristics of testers and non-testers do not indicate any clear biases that would 
necessarily undermine the comparisons. 
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Table 4-2. Characteristics of DC Public School Free or Reduced-Price Lunch 

Program Students, Program Applicants Versus Nonapplicants: Spring 2004 



Characteristic 


Applicants 


DCPS 

Sample 


Difference 


Baseline Test Scores 1 








Average Reading Percentile 


40.2 


36.4 


3 9 ** 


Percent missing 


25 


26 




Average Mathematics Percentile 


46.7 


43.0 


2 7** 


Percent missing 


25 


24 




Percent in Special Education 


17 


15 


1 


Percent missing 


24 


22 




Percent, by Race 








African American 


93 


88 


5** 


Hispanic 


6 


10 


-5** 


Other race 2 


2 


2 


0 


Percent missing 


0 


0 




Percent, by Gender 








Female 


51 


51 


0 


Percent missing 


0 


0 




Percent Participating in Free/Reduced- 








price Lunch Program 


100 


100 


0 


Percent missing 


0 


0 




Sample size 


894 


44,740 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in tenns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes students who were identified as white, Asian, American Indian, or Alaska Native. 

NOTE: Detail may not sum to totals because of rounding. Applicant sample includes all eligible applicants identified in the 

DCPS database that were participating in the free and reduced price lunch program. 

SOURCE: Accountability testing database for District of Columbia public and charter schools, DCPS 
Office of Communications and Public Information. 
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This most detailed comparison of applicants and nonapplicants is nearly identical to the other analyses but 
provides some additional information (Table 4-3): 

• Although program applicants have higher reading and mathematics achievement than non-SINI 
public school students who are similar in terms of income and grade level, these differences are 
only statistically significant among elementary students. 55 

• There are no statistically significant differences in the rates of special education enrollment 
between applicants and similar nonapplicants in any grade band. 

• The higher rates of African American applicants and lower rates of applicants of other races are 
evident at both the elementary and high school levels but not at the middle school level. 

• Again, by design, the applicant group is similar to the nonapplicant comparison group in all three 
grade bands regarding the proportions that are enrolled in the FRL program. 



Public School Participants Compared to Economically Disadvantaged Nonapplicant DCPS Students, by 
Grade Bands 

The legislation requires that the evaluation include a comparison between participating eligible students 
and DCPS students in the same grades, an analysis we refer to as performance reporting. Participants 
differ from the group of public school applicants above, in that participants: (1) include students who 
were already attending private school when they applied to the program, and (2) exclude students who did 
not receive scholarships as part of the lottery for those entering grades 6 through 12 (the impact sample’s 
control group). Unfortunately, at baseline (before the program begins), it is not possible to compare the 
prior academic achievement of scholarship recipients who were already enrolled in private schools 
because no test data comparable to that of the DCPS accountability assessment is available from private 
schools. 56 

Because public school participants are such a large subset of public applicants, comparing either group to 
similarly economically disadvantaged DCPS students in the same grade levels yields consistent 
similarities and differences (Table 4-4): 

• Program participants in grades K to 5 score somewhat higher in reading and mathematics than 
comparable nonapplicants. They also are more likely to be African American and less likely to 
be Hispanic than are nonapplicants. These results are identical to the K to 5 comparisons of 
eligible public school applicants to DCPS nonapplicants in Table 4-3 because all eligible public 
school applicants in grades K to 5 automatically became program participants. 57 



Test scores were missing from about one-quarter of the sample, so these results should be interpreted with caution. However, 
as discussed in footnote 47, the background characteristics of testers and non-testers do not indicate any clear biases that would 
necessarily undermine the comparisons. 

56 In addition, DCPS measures other characteristics, such as race/ethnicity and special education status, differently than does the 
baseline survey the evaluation team administered 

57 

Test scores were missing from about one-quarter of the sample, so these results should be interpreted with caution. However, 
as discussed in footnote 47, the background characteristics of testers and non-testers do not indicate any clear biases that would 
necessarily undermine the comparisons. 
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Table 4-3. Characteristics of DC Public School Free or Reduced-Price Lunch Students by Grade Band, Program Applicants Versus 
Nonapplicants: Spring 2004 





Grades K-5 


Grades 6-8 


Grades 9-12 






DCPS 






DCPS 






DCPS 




Characteristic 


Applicants 


Sample 


Difference 


Applicants 


Sample 


Difference 


Applicants 


Sample 


Difference 


Baseline Test Scores 1 




















Average Reading Percentile 


46.5 


42.6 


4.0* 


38.0 


36.4 


1.5 


29.7 


28.8 


0.8 


Percent missing 


42 


44 




3 


5 




3 


14 




Average Mathematics Percentile 


51.2 


47.8 


3.5* 


43.8 


42.3 


1.5 


41.7 


37.9 


3.8 


Percent missing 


41 


42 




2 


5 




3 


13 




Percent in Special Education 


17 


13 


3 


18 


17 


1 


14 


16 


-2 


Percent missing 


41 


41 




2 


3 




2 


9 




Percent, by Race 




















African American 


94 


87 


7 ** 


88 


90 


-1 


95 


89 


6* 


Other race 2 


6 


13 




12 


10 


1 


5 


11 


-6* 


Percent missing 


0 


0 




0 


0 




0 


1 




Percent, by Gender 




















Female 


52 


51 


i 


51 


51 


0 


46 


52 


-6 


Percent missing 


0 


0 




0 


0 




0 


1 




Percent Participating in Free/Reduced- 
price Lunch Program 


100 


100 


0 


100 


100 


0 


100 


100 


0 


Percent missing 


0 


0 




0 


0 




0 


0 




Sample size 


509 


20,893 




259 


12,364 




126 


11,483 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in tenns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes students who were identified as Hispanic, white, Asian, American Indian, or Alaska Native. Because of small cell sizes, Hispanic was combined with 
other race in this table. 

NOTE: Detail may not sum to totals because of rounding. Applicant samples include all eligible applicants identified in the DCPS database that were participating in the free 

or reduced-price lunch program. 

SOURCE: Accountability testing database for District of Columbia public and charter schools, DCPS Office of Communications and Public Information. 
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Table 4-4. Characteristics of DC Public School Free or Reduced-Price Lunch Students by Grade Band, Program Participants Versus 

Nonapplicants: Spring 2004 





Grades K-5 


Grades 6-8 


Grades 9-12 






DCPS 






DCPS 






DCPS 




Characteristic 


Participants 


Sample 


Difference 


Participants 


Sample 


Difference 


Participants 


Sample 


Difference 


Baseline Test Scores 1 




















Average Reading Percentile 


46.5 


42.6 


4.0* 


37.7 


36.4 


1.3 


24.0 


28.8 


-4.8 


Percent missing 


42 


44 




3 


5 




4 


14 




Average Mathematics Percentile 


51.2 


47.8 


3.5* 


43.0 


42.3 


0.7 


41.0 


37.9 


3.1 


Percent missing 


41 


42 




1 


5 




4 


13 




Percent in Special Education 


17 


13 


3 


17 


17 


0 


19 


16 


2 


Percent missing 


41 


41 




1 


3 




2 


9 




Percent, by Race 




















African American 


94 


87 


y** 


88 


90 


-2 


95 


89 


6 


Hispanic 


5 


12 




9 


9 


1 


0 


9 


_9* 


Other race 2 


1 


2 


-i 


2 


1 


1 


5 


2 


4* 


Percent missing 


0 


0 




0 


0 




0 


1 




Percent, by Gender 




















Female 


52 


51 


i 


50 


51 


-2 


45 


52 


-6 


Percent missing 


0 


0 




0 


0 




0 


1 




Percent Participating in 
Free/Reduced-price Lunch Program 


100 


100 


0 


100 


100 


0 


100 


100 


0 


Percent missing 


0 


0 




0 


0 




0 


0 




Sample size 


509 


20,893 




201 


12,364 




55 


11,483 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in terns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes students who were identified as white, Asian, American Indian, or Alaska Native. 

NOTE: Detail may not sum to totals because of rounding. Participants samples differ from previous samples because they are limited to scholarship recipients identified in the 

DCPS database that were participating in the free or reduced-price lunch program. 

SOURCE: Accountability testing database for District of Columbia public and charter schools, DCPS Office of Communications and Public Information. 




• The only statistically significant differences between public school program participants and 
nonapplicants in grades 6 to 12 involve student race and ethnicity. High school participants are 
less likely to be Hispanic and more likely to be a race other than African American or Hispanic 
than are nonapplicants. 

• Within all three grade bands, program participants are statistically similar to DCPS nonapplicants 
in their special education status, gender, and (by design) eligibility for free or reduced-price 
lunch. 



4.2 Applicants in the Impact and Non-Impact Samples 

While it is important to examine who applies to the DC Opportunity Scholarship Program, for the 
evaluation it is equally critical to ensure that the applicants who will be the focus of the impact analysis — 
the treatment and control groups randomly assigned as part of the lottery — are similar prior to the 
beginning of the program. It is on this similarity between the groups, not only in characteristics easily 
measured but also in those not observed, that the scientific benefits of the RCT approach rests. As 
described in Chapters 1 and 2, the impact sample consists of public school applicants who were not from 
schools in need of improvement (non-SINI) and who were entering the oversubscribed grades of 6 
through 12. 

However, because the impact sample, at least in the first year of the program, is only a subset of all 
program applicants, it is also useful to review how the characteristics of the impact sample differ from the 
characteristics of students who will not be included in the analysis of program effectiveness (or what we 
will call the “non-impact sample” of applicants). 58 The existence of many significant differences between 
the impact and non-impact samples might limit one’s ability to extrapolate the results of the impact 
analysis to measuring the program impacts as a whole. 59 

Because grade level can account for significant differences in characteristics and the impact sample 
includes only students entering grades 6 through 12, the comparisons are limited to those grades and 
suggest that (Table 4-5): 

• The treatment and control groups are statistically similar on important educational and family 
background measures that could affect outcomes, such as baseline test scores, racial/ethnic 
composition, mother’s average years of school, and family income. Although the two groups are 
not identical, the differences that exist are within the range of what we might expect due to mere 
chance. None of the differences are statistically significant. 



As described in Table 3-3, the non-impact sample consists of 1,356 eligible applicants. A total of 1,067 are scholarship 
recipients - 851 of whom were public school applicants who were automatically awarded scholarships and 216 of whom were 
private school applicants. The non-impact sample also includes 289 scholarship nonrecipients who were private school 
applicants. 

59 It is not appropriate, statistically, to assume that the impact analysis results would apply to the student groups not included in 
the impact sample. In particular, applicants already enrolled in private schools, excluded from the impact sample, are likely to 
be quite different than public school applicants. 
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Table 4-5. Characteristics of the Impact Sample and Non-Impact Sample for Applicants 
Entering Grades 6 to 12, Spring 2004 



Characteristics 


Treatment 


Control 


Difference 




Non- 

impact 

Sample 


Difference 


Baseline Test Scores 1 














Average Reading Percentile 














Grades 6-8 


39.6 


42.7 


-3.1 


40.3 


35.1 


5.2 


Grades 9-12 


32.9 


35.6 


-2.7 


34.9 


19.5 


15.3** 


Percent missing 


17 


22 




19 


84 




Average Mathematics Percentile 














Grades 6-8 


44.2 


48.1 


-4.0 


45.1 


42.0 


3.1 


Grades 9-12 


47.6 


43.8 


3.8 


44.9 


36.3 


8.6 


Percent missing 


16 


22 




18 


85 




Percent of Students with a 


18 


23 


-5 


20 


12 


7 ** 


Learning or Physical Disability 














Percent missing( average ) 


6 


7 




6 


3 




Percent, by Race 














African American 


93 


94 


-1 


93 


90 


3 


Other race 2 


7 


7 


0 


7 


10 


-2 


Percent missing 


5 


6 




5 


10 




Percent Hispanic (any race) 


9 


7 


2 


8 


16 


_g** 


Percent missing 


4 


7 




5 


4 




Mother’s Average Years of 


12.6 


12.8 


-.2 


12.7 


12.8 


-.2 


Schooling 














Percent missing 


17 


16 




17 


15 




Average Family Income 


$18,767 


$19,029 


-$262 


$18,870 


$19,334 


-$464 


Percent missing 


0 


0 




0 


0 




Percent of Students whose 


49 


54 


-5 


51 


48 


4 


Mothers are Employed Full 














Time 














Percent missing 


19 


19 




19 


17 




Sample size 


299 


193 




492 


321 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in terns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes respondents who were identified as white, Asian, American Indian, Alaska Native, Pacific 
Islander, or multiracial. Respondents classified as “multiracial” if more than one race category was selected. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: Eligible Applicant Database. Test-scores obtained from the accountability testing database for District of Columbia 
public and charter schools, DCPS Office of Communications and Public Information. 
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• There are a few significant differences between the impact sample as a whole and the non-impact 
sample, at least for those entering grades 6 through 12. While the extent of missing baseline 
achievement data for the non-impact sample precludes a reliable comparison of test scores, 
members of the impact sample are more likely than members of the non-impact sample to report 
having a learning or physical disability (20 percent vs. 12 percent) but are less likely to be of 
Hispanic ethnicity (8 percent vs. 16 percent). 

These results give us confidence that the random assignment process worked as designed to generate two 
comparable subgroups for the impact analysis. However, the impact analysis sample does not appear to 
be comparable to the sample of applicants who will not be included in the rigorous part of the evaluation 
on at least a few dimensions. 



4.3 Eligible Applicants by Type of Previous School 

The differences between evaluation subsamples underscore that the program applicants are a diverse 
group, and these differences may play a role in the extent to which the program is effective and for which 
students. In particular, applicants are likely to vary somewhat in their educational and background 
characteristics depending on the type of school that they attended previously — SINI, non-SINI public 
schools, charter schools, and private schools. 60 

As described in Chapter 3, the eligible applicants were drawn primarily from public schools that were not 
designated in need of improvement under NCLB at the time of application. 61 A total of 15 schools, 
representing 9 percent of all DCPS, had been designated as SINI as of the program application period. 
Seventy-nine students, or 4 percent of all eligible applicants, were drawn from the schools designated as 
SINI during the 2003-04 school year. A total of 993 students, or 54 percent of all eligible applicants, 
were drawn from non-SINI public schools. Public charter schools generated 258 applicants, or 14 percent 
of the total, with private schools accounting for the remaining 518 eligible applicants (28 percent). 
Detailed information on all applicants was collected as part of the application process and analyzed for 
differences and similarities by type of school previously attended. As a way of summarizing, in these 
analyses an average for a school type subgroup was designated as significant if it differs statistically from 
the average for all applicants with that particular subgroup excluded. 62 



60 The data presented in this chapter were drawn almost exclusively from the application form and baseline survey, with the 
exception of test-score data taken from the DCPS database. 

61 The numbers of eligible applicants by subgroup differ somewhat in this chapter from the totals used for the priority-group 
classifications described in Chapters 2 and 3. First, the 258 applicants from charter schools are a significant component of the 
non-SINI public group, and, thus, were separated out from that group for the purposes of this analysis. Second, 13 applicants 
were originally classified as attending non-SINI public schools based on parental responses to the school type and name of 
school questions in the baseline survey. In all 13 cases, the parent indicated that the child was attending a public school and 
provided either no school name (4 cases) or a school name that did not obviously signify the type of school (9 cases — 4 of 
which were preschools). Further investigations by the technical advising team indicated that these 13 students were attending 
private schools or private preschools, and their school type classification was corrected accordingly. 

62 Specifically, for each characteristic, each subgroup (e.g., SINI Public, Non-SINI Public, Charter, Private) average was 
compared with the population average excluding that subgroup, using a t test to determine if the subgroup score was 
statistically different from the collective members of the other subgroups. For example, the average reading score of 19.5 
NPRs for SINI public applicants entering grades 9 to 12 was significantly different from the average for all other applicants 
excluding those from SINI public schools. This approach was selected instead of conducting multiple pair-wise comparisons 
because it directly and efficiently addresses the question “Is this subgroup average different from the rest?” The t test was 
used instead of a Z test because the t test is a more powerful statistical test, which is an important concern when subgroups are 
small (e.g., the SINI public subgroup). 
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Educational Characteristics of Eligible Applicants 



We examined the grade-level distributions, average test scores, disability status, and English proficiency 
for all applicants by previous type of school. Pre-application test score data were only available for public 
and charter school applicants who completed the DCPS accountability tests in spring 2004. Comparable 
test data were not available for private school applicants. But other significant findings include the 
following: 

• More than half of all applicants (55 percent) were at the elementary level (Table 4-6). SIN1 public 
schools were less likely to have elementary school program applicants and more likely to have 
high school applicants than other types of schools, although this largely reflects the proportion of 
SINI-designated schools that were high schools (7 of 15 ). 63 Non-SINI public schools, which had 
the majority of program applicants, were substantially more likely than other schools to send 
elementary-level applicants and less likely to generate high school applicants. 

• Overall, applicants tended to score somewhat higher in mathematics than in reading and higher in 
the younger grades than in the older grades (Table 4-7). At the high school level, applicants from 
SINI public schools scored substantially lower in reading — and applicants from non-SINl public 
schools significantly higher — than applicants from non-SINl public schools. The test scores for 
the remaining grade-band and school-typesubgroups in reading, and all of the subgroup averages 
in mathematics, are statistically similar to the applicant norms . 64 

Table 4-6. Percentage of Applicants in Various Grade Bands, By Type of School 
Previously Attended: Spring 2004 



Forecasted Grade Level 
Fall 2004 


Total 


SINI 

Public 


Non-SINI 

Public 


Charter 


Private 


Kindergarten-5 


55 


32** 


66 ** 


45* 


47 ** 


6-8 


27 


26 


25 


33 


31 


9-12 


16 


44 ** 


io** 


20 


22 


Total 


100 


100 


100 


100 


100 


Sample size 


1,848 


79 


993 


258 


518 



* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



SOURCE: Eligible Applicant Database. 



63 Seven of the 15 SINI schools were high schools, and high school students were significantly less likely to apply for the 
program than were students in the elementary and middle school grades. 

64 Test scores were missing from all of the private school applicants and one-quarter to one-half of the members of the other 
subgroups, so these results should be interpreted with great caution. 
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• For the total population of eligible applicants, 14 percent have a learning or physical disability, 
according to parental reports (Table 4-7). Applicants from non-SINI public schools are slightly 
more likely to report a learning or physical disability, whereas private school applicants are 
somewhat less likely to report such a disability. 65 



Table 4-7. Educational Characteristics of Eligible Applicants by School Type: Spring 2004 



Characteristics 


Total 


Public SINI 


Non-SINI 

Public 


Charter 


Private 


Baseline Test Scores 1 

Average Reading Percentile 












Grades K-5 


47.0 


49.9 


47.5 


43.8 


NA 


Grades 6-8 


39.9 


35.1 


39.9 


41.3 


NA 


Grades 9-12 


31.7 


19.5** 


38.0** 


27.9 


NA 


Percent missing 


57 


22 


42 


38 


100 


Average Mathematics 
Percentile 












Grades K-5 


51.1 


67.1 


51.0 


48.5 


NA 


Grades 6-8 


44.8 


42.0 


44.5 


46.7 


NA 2 


Grades 9-12 


43.1 


36.3 


46.0 


42.4 


NA 


Percent missing 


56 


23 


41 


38 


100 


Percent with a Learning or 
Physical Disability 














14 


16 


16* 


16 


10** 


Percent missing (average) 


7 


11 


9 


5 


6 


Sample size 


1,848 


79 


993 


258 


518 



* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in tenns of National Percentile Ranks, with 50 as the median score. 

SOURCE: Eligible Applicant Database. Test scores obtained from the accountability testing database for District of 
Columbia public and charter schools, DCPS Office of Communications and Public Information. 



Background Characteristics of Eligible Applicants 



We also collected information about the demographics and socioeconomic status of applicants: 

• The overwhelming majority of eligible applicants to the program, 95 percent, are African 
American (Table 4-8). Private school applicants were somewhat less likely to identify their race 



65 Both the size of the difference and the size of the samples being compared factor into the determination of statistical 
significance. As a result, even though the subgroup of SINI public applicants reported the highest rate of learning or physical 
disabilities (16 percent), that level does not differ statistically from the rates of students from the other subgroups because the 
sample of SINI public applicants is so small — -just 79 students. 
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as African American (92 percent), whereas applicants from charter schools were somewhat more 
likely to do so (98 percent). 66 

• The average family income of applicants in 2003 was $18,742. The family incomes of SINI 
public and non-SINI public applicants averaged slightly less than the other groups, whereas the 
incomes of private school applicants were somewhat higher than the norm. 

• Mothers’ education is also considered an important indicator of socioeconomic status. The 
mothers of applicants averaged almost 13 years of postkindergarten education. 67 Mothers of 
applicants from non-SINI public schools averaged slightly less formal education than those in the 
other subgroups, and the mothers of private school applicants averaged somewhat more formal 
education (13.2 years). 

• Nearly half of the mothers of applicants (47 percent) were employed full time at the time of 
application. Applicants from non-SINI public schools were somewhat less likely to have mothers 
who worked full time (45 percent), and charter school applicants were significantly more likely 
(54 percent). About one-fifth of the program applications indicated that the student’s mother is 
married, a percentage that did not vary significantly among the various applicant groups. 

Additional information about such baseline survey items as parental views of and satisfaction with their 
child’s school are presented in Appendix C. 



4.4 Scholarship Users and Nonusers 

Some people who are offered the benefit of a new government program inevitably decide not to avail 
themselves of it. This has been the case for both government-financed and private-funded school choice 
programs in the United States. As discussed in Chapter 3, the initial scholarship usage rate for the DC 
Opportunity Scholarship Program was 75 percent, though the usage rate for members of the impact 
subsample was somewhat lower, at 62 percent. 

Those earlier studies of school choice programs found that students who actually use scholarships tend to 
differ modestly from students who receive scholarships but do not use them. 68 Most, but not all, 
differences suggested that scholarship users are somewhat more advantaged academically and in terms of 
family background than students who decline to use their scholarships. 



66 Again, the proportion of applicants who are African American was highest among the SINI public subgroup, but the small size 
of that subgroup means that we cannot know with confidence that it is significantly different from the norm in that regard. 

67 Comparing the applicant data to census data for the DC for all adults ages 25 to 64 years old in households that are at or below 
185 percent of poverty in the District indicates that the mothers of program applicants have higher levels of educational 
attainment. This may or may not suggest that applicant families are relatively advantaged, because the census numbers include 
men, whose educational attainment tends to be lower than that of women. Although the baseline application and survey 
requested information about the applicant’s birth father, only 10 to 15 percent of applications included responses to those 
questions. 

68 William G. Howell, “Dynamic Selection Effects in Means-Tested, Urban School Voucher Programs,” Journal of Policy 
Analysis and Management 23, No. 2, Spring 2004, pp. 225-250; William G. Howell and Paul E. Peterson, with Patrick J. Wolf 
and David E. Campbell, The Education Gap: Vouchers and Urban Schools (Washington, DC: Brookings, 2002), pp. 65-80; 
John F. Witte, The Market Approach to Education (Princeton, 2000), pp. 60-61. 
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Table 4-8. Demographic Characteristics of Eligible Applicants by School Type: Spring 2004 



Characteristics 


Total 


Public 

SINI 


Non-SINI 

Public 


Charter 


Private 


Percent, by Race 












African American 


95 


96 


96 


98* 


92** 


Other race 1 


5 


4 


4 


2 * 


g** 


Percent missing 


9 


1 


8 


5 


13 


Percent of Students whose 
Mothers have the Following 
Level of Education 












No high school diploma 


13 


20 


15** 


9 * 


10 * 


High school diploma/GED 


29 


27 


32 


30 


24 


Some college, no degree 


32 


32 


31 


37 


32 


Associates degree or higher 


25 


22 


21 * 


23 


33* 


Percent missing 


13 


16 


13 


18 


12 


Mother’s Average Years of 


12.77 


12.62 


12.56** 


12.85 


13.15** 


Schooling 












Percent missing 


13 


16 


13 


17 


12 


Average Family Income 


$18,742 


$16,033* 


$18,021* 


$19,328 


$20,246** 


Percent missing 


0 


0 


0 


0 


0 


Percent of Students whose 


47 


43 


45* 


54* 


49 


Mothers are Employed Full 
Time 












Percent missing 


16 


20 


16 


18 


14 


Percent of Students whose 


19 


21 


19 


16 


22 


Mothers are Married 












Percent missing 


12 


14 


11 


13 


13 


Sample size 


1,848 


79 


993 


258 


518 



* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 “Other race” includes respondents who were identified as white, Asian, American Indian, Alaska Native, Pacific 
Islander, or multiracial. Respondents classified as “multiracial” if more than one race category was selected. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: Eligible Applicant Database. 
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Differences between scholarship users and nonusers in the impact sample are a particular concern, since it 
could affect the ability of the evaluation to generate reliable assessments of program impact. Fortunately, 
the scholarship user and nonuser subgroups within the impact are similar in almost all key respects (Table 
4-9, Impact Sample columns). Although users are less likely than nonusers to report having a learning or 
physical disability (12 percent versus 27 percent), users are statistically comparable to nonusers regarding 
a number of important academic and family background factors including baseline test scores, race and 
ethnicity, income, and mother’s years of schooling and employment status. 

A comparison of the user and nonuser subgroups in the larger Program Sample reveals a few more 
statistically significant differences than in the Impact Sample. Among all the scholarship recipients in the 
DC Opportunity Scholarship Program, the 1,027 who actually used the scholarship to enroll in a preferred 
private school differed from the 339 nonusers in several important respects (Table 4-9, Program Sample 
columns). The notable differences between scholarship users and nonusers in the Program Sample 
included the following (see Appendix D for the full set of comparisons): 

• Users entering grades K to 5 scored somewhat higher in reading (8.8 NPR) and mathematics (9.7 
NRP) than nonusers. However, this result should be interpreted with extreme caution, as baseline 
test scores were available for less than half of the students. 

• Nine percent of users reported having a learning or physical disability compared to 29 percent of 
nonusers. 

• Mothers of scholarship users reported .2 more years of education than mothers of nonusers. 

There were no statistically significant differences between scholarship users and nonusers in the Program 
Sample regarding the following: 



• Test-score performance for students in grades 6 to 8 or 9 to 12, 

• Student’s race or ethnicity, 

• Family income, or 

• Mother’s employment. 

It is impossible to know at this point if the sizeable and consistent difference between scholarship users 
and nonusers regarding learning or physical disabilities is the result of parental decisions regarding what 
is best for their child or limitations in the ability of participating private schools to accommodate the 
special educational challenges that some scholarship students face. At the same time, there is no evidence 
in the data that scholarship recipients of differing races, ethnicities, or family incomes are differentially 
willing or able to use an Opportunity Scholarship to switch to a private school of their parent’s choosing. 



Page 48 





Table 4-9. Educational and Demographic Characteristics of the Users and Nonusers in 
the Impact Sample and Program Sample, Spring 2004 





Impact Sample 


Program Sample 


Characteristics 


Users 


Nonusers 


Difference 


Users 


Nonusers 


Difference 


Baseline Test Scores 1 














Average Reading Percentile 














Grades K-5 


NA 


NA 


NA 


49.1 


40.3 


8.8** 


Grades 6-8 


39.9 


38.9 


1.1 


39.8 


38.1 


1.8 


Grades 9-12 


39.6 


27.7 


11.9 


30.0 


23.6 


6.4 


Percent missing (average) 


16 


20 




56 


41 




Average Mathematics Percentile 














Grades K-5 


NA 


NA 


NA 


53.4 


43.8 


Q 7 ** 


Grades 6-8 


44.8 


43.0 


1.8 


44.9 


42.3 


2.7 


Grades 9-12 


49.4 


46.1 


3.4 


40.3 


43.8 


-3.5 


Percent missing (average) 


14 


19 




56 


40 




Percent of Students with a Learning 
or Physical Disability 


12 


27 


-15** 


9 


29 


-20** 


Percent missing (average) 


6 


5 




6 


8 




Percent, by Race 














African American 


92 


96 


-4 


94 


96 


-2 


Other race 2 


8 


4 


4 


6 


4 


2 


Percent missing 


7 


5 




6 


6 




Percent Hispanic (any race) 


10 


7 


2 


7 


7 


1 


Percent missing 


3 


5 




4 


7 




Average Family Income 


$18,342 


$19,457 


-$1,115 


$18,652 


$17,841 


$812 


Percent missing 


0 


0 




0 


0 




Mother’s Average Years of 
Schooling 


12.5 


12.7 


-0.3 


12.7 


12.5 


.2* 


Percent missing 


14 


23 




11 


19 




Percent of Students whose Mothers 
are Employed Full Time 


49 


50 


-1 


46 


44 


2 


Percent missing 


16 


25 




13 


23 




Sample size 


185 


114 




1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in tenns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes respondents who were identified as white, Asian, American Indian, Alaska Native, Pacific 
Islander, or multiracial. Respondents classified as “multiracial” if more than one race category was selected. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: Eligible Applicant Database. Test-scores obtained from the accountability testing database for District of Columbia 
public and charter schools, DCPS Office of Communications and Public Information. 
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4.5 Applicant Response Rates Among Public and Private Schools 

In addition to examining the impact on students, the law also calls for the evaluation to examine the 
impact of the program on schools in the District. Such “systemic effects” could take place if significant 
percentages of students in the public school system or in specific schools apply for, receive, and use 
scholarships to transfer to private schools. With regard to the public schools, the DC Opportunity 
Scholarship Program could have either positive or negative effects. One theoretical argument suggests 
that scholarship programs will divert funding and the most motivated students from public schools to 
private schools, leaving the public school system with fewer resources with which to educate the 
remaining student population. 69 Another theory is that schools behave in a manner similar to firms and 
will respond to competition by becoming more efficient. 70 In the case of schools, it is the risk of losing 
students and subsequently funding that may provide an impetus for public schools to provide additional 
services and produce better student outcomes. Private schools in jurisdictions with greater school choice 
may face similar incentives to improve or expand in order to retain as many of their current students and 
attract more. 

In practice, two features of choice programs are required if effects on public schools are to be observed. 
First, the threat of losing students to the choice program must be significant. 71 It is important to note that 
the threat need not be that students actually take the scholarship (at least initially) but that the program is 
available to a large share of the student population. The second feature is that the loss of students to the 
choice program also entails a loss of funding. However, according to a Memorandum of Understanding 
between the Office of the DC Mayor and DCPS, public schools that lose students as a result of the 
Opportunity Scholarship program will be reimbursed for any lost resources, thereby eliminating any fiscal 
impact of the program on specific public schools, at least in the short run. Still, an analysis of the 
response of public schools to the scholarship program is both mandated and of policy interest. 



Public School Application and Use Rates 

To provide a foundation for a later analysis of the effects of the DC Opportunity Scholarship Program on 
DC schools, we first need to describe the extent to which public schools have so far been affected by 
program applications and scholarship users — an indicator of the seriousness of the pressure that the public 
schools may face. There are 197 public schools in Washington, DC, including all DCPS schools and all 
public charter schools. 72 We ranked those schools based on the proportion of students in the school who 
applied for the program or used a scholarship to transfer out of the school, from 0 percent to at least 5 
percent (Table 4-10): 



69 Henry M. Levin, Privatizing Education: Can the Marketplace Deliver Choice, Efficiency, Equity, and Social Cohesion? 
(Boulder, CO: Westview, 2001), 6. 

70 See especially Caroline M. Hoxby, “School Choice and School Competition: Evidence from the United States,” Swedish 
Economic Policy Review 10(2003): 11-67. 

71 For example, see David L. Armour and Brett M. Peiser, “Interdistrict Choice in Massachusetts,” in Paul E. Peterson and Bryan 
C. Hassel, eds., Learning From School Choice (Washington, DC: Brookings, 1998), 157-186. 

72 The list of schools was obtained from the web sites of DCPS and the DC Public Charter School Board. 
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Table 4-10. Public School-Level Scholarship Application and Use Rates: 
Spring 2004 and Fall 2004 



Percent of 
Student Body 


Applied for tl 


le Program 


Used a Scholarship 


Number of 
Schools 


Percent of 
Schools 


Number of 
Schools 


Percent of 
Schools 


0 


25 


13 


51 


26 


0.1 - 1.0 


47 


24 


63 


32 


1.1 -2.0 


56 


28 


47 


24 


2.1 -3.0 


38 


19 


21 


11 


3.1 -4.0 


12 


6 


12 


6 


4.1 - 


19 


11 


3 


2 


Total 


197 


100 


197 


100 



NOTE: Detail may not sum to totals because of rounding. 



SOURCES: Application and usage numbers by school generated from the Applicant Database and 
WSF Placement Database. Enrollment figures for DCPS are from the 2002-23 school 
year and were obtained from Membership in the District of Columbia Public Schools 
by School and Grade, October 7, 2003, available on the DCPS web site, 
www.kl2.dc.us/dcps/data/enrollment/membership-Oct.703_official_.pdf. Enrollment 
figures for DC public charter schools chartered by the District Board of Education are 
from the 2002-03 school year and were obtained from the Common Core of Data, 

National Center for Education Statistics. Enrollment figures for the DC public charter 
schools chartered by the DC Public Charter School Board are from the 2003-04 school 
year and were obtained from the Board’s web site at www.dcpubliccharter.com. 

• Over one-quarter of the public schools in the District experienced no student losses due to the 
program. 

• Another 56 percent of DCPS schools had program-related transfers out that totaled less than 2 
percent of their student populations. 

• Seventeen percent of District schools lost about 2 to 4 percent of their students. 

• Finally, 2 percent of the public schools in the District experienced more significant student 
transfers of over 4 percent under the program. 

For now, it appears that very few public schools in the District have experienced a significant loss of 
students to the DC Opportunity Scholarship Program. 



Private School Reliance on DC Choice Program 

Much of the discussion surrounding K to 12 scholarship programs revolves around the composition of 
students entering private schools. Schools participating in choice programs make a second, perhaps 
equally important, set of decisions about the number of students to admit. The larger the share of entering 
students that come through these programs, the greater is the investment of the private school in the 
program and the potential exposure to the challenges of assimilating a sizable population of new private 
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school students who are using scholarships. Moreover, private schools with a large proportion of their 
student body comprised of Opportunity Scholarship students may become relatively dependent on the 
program for the necessary resources to operate. Or they may choose to continue to expand their facilities 
in order to serve additional scholarship students. 

Fifty-eight DC private schools participated in the DC Opportunity Scholarship Program in the first year. 
Of those, reliable enrollment data were available for 54 schools. Our analysis indicates that, for most of 
the participating private schools, students using DC Choice Opportunity Scholarships make up a 
significant share of their enrollments (Table 4-1 1) 73 : 

• In 28 percent of the schools over 20 percent of their students are using Opportunity Scholarships 
this year, including 9 percent of the schools with over half of their students using program 
scholarships. 

• In an additional 37 percent of the private schools, scholarship students make up between 5 and 20 
percent of their student population this school year. 

• Slightly more than one-third will have student bodies comprised of less than 5 percent program 
scholarship students in 2004-05. 

The extent of student losses and gains due to the program suggest that private schools in the District may 
be more affected by the initial implementation of the DC Opportunity Scholarship Program than are DC 
public schools at this point in time. 

Table 4-11. Private School Reliance on Scholarship Program: 

Fall 2004 



Percent of Student Body 


Schools with Sc 


holarship Users 


Number of 
Schools 


Percent of 
Schools 


0 


4 


7 


O 

I/S 

1 

o 


15 


28 


5.1 - 10.0 


6 


11 


10.1 - 15.0 


7 


13 


15.1-20.0 


7 


13 


20.1 -33.3 


7 


13 


33.4-50.0 


3 


6 


50.1 - 


5 


9 


Total 


54 


100 



NOTE: Detail may not sum to totals because of rounding. 

SOURCE: Scholarship users drawn from the WSF Placement Database. Private school 
enrollments are for the 2003-04 school year and are drawn from the WSF 
DC Opportunity Scholarship Directory of Participating Schools. 



73 Students using DC Opportunity Scholarships at private schools include both students who transferred to those schools from 
public schools and students who were already attending the private school before they applied for and received a scholarship 
through the lottery. 
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Appendix A 




Appendix A: Analysis of Study Power 



To ensure that the experimental evaluation of program impact will produce reliable findings, the sample size 
must be large enough to enable the analysis to answer the study’s central questions and to measure program 
effects that are large enough to be both meaningful in students’ lives and relevant to policy debates about the 
efficacy of educational interventions. The ability of a study to do so is a function of the study’s precision or 
“power.” 

Minimum detectable effects are a simple way to express the statistical precision of an impact study 
design. Intuitively, a minimum detectable effect is the smallest program impact that could be measured 
with confidence given random sampling and statistical estimation error . 74 For example, from a benefit- 
cost perspective one might ask whether a proposed sample could reliably detect the smallest impact 
needed for a program to “break even” (that is, produce benefits equal to costs). One would want a sample 
that was large enough to ensure that an estimated impact around this “break-even” point was a reliable 
indicator of the program’s true impact and not just due to chance variation. A smaller sample might 
enable the study to detect only impacts that are well above this break-even point and thus very difficult or 
costly to attain. A study that used a sample of this size would therefore miss an opportunity to produce 
reliable estimates of policy-relevant impacts. Hence, it would be “under-powered” statistically. 

The goals of statistical power analysis and sample size estimation are to determine how large a sample is 
needed to make accurate and reliable statistical judgments; and how likely a statistical test will detect 
effects of a given magnitude. 

To determine the sample size required for the scholarship program, we use the following information. 
Suppose the statistical model is given by: 

T=yi +Y2P + Y3X+S 

where T= test score, 

P= treatment status, and 

X demographics, baseline score and time controls. 
y 2 is the treatment effect 

The null hypothesis, i.e. the initial assumption that the treatment has no effect, is 72 = 0 . The power is the 
probability of rejecting this null if the treatment actually has an effect. 

Let: 

a be the statistical significance level, set equal to 0.05 (i.e. 95 percent confidence). 

(1-P) be the power of the test (acceptable range of 80-90 percent). 
n T be the sample size for the treatment sample. 
n c be the sample size for the control sample. 

< 7 be the standard deviation for an outcome of interest, in this case, test scores. 

/?be the correlation between a given student’s test scores at baseline and outcome year 1 . 



74 We define a minimum detectable effect as the smallest true program impact that would have an 80 percent chance of being 
detected (have 80 percent power) using a two-tail hypothesis test at the 0.05 level of statistical significance. We use a two-tail 
test because it is conceivable that the scholarship program could have either a negative or positive effect on test scores, even 
though the policy question is about improved test scores. Clustering of students may also affect the sample size calculations, 
however the extent of that clustering is unknown at this time. When the data is analyzed, it may be necessary to increase the 
sample needed or the size of the effect that can be detected. 



Page A-l 





Estimated Sample Size Required For Desired Power Analyzing Test Scores in Outcome Years 1 and 3 

Although the evaluation will analyze outcomes such as test scores in every year post baseline, for 
simplicity, we present the power analysis numbers for two representative years - the first and third 
outcome years. Central to analytic power is the sample size of study participants who actually provide 
outcome information in a given year. Thus, this power analysis factors in expected study attrition over 
time. It also takes account of the correlation between baseline test scores and outcome test scores. By 
including baseline test scores in the statistical estimation of outcome test scores, analysts make the 
estimation of the impact of the treatment on the outcome more precise, thus increasing power. Since 
baseline test scores were obtained for most but not all members of the impact sample, the power analysis 
is conducted under conditions of the absence of baseline test scores and the presence of baseline test 
scores. Finally, all else equal, power is greatest when the treatment and control groups are the same size. 
It is unlikely that this condition will be met in practice. Therefore, the sample size estimations are made 
under the condition of balanced treatment and control groups and the more likely scenario of twice as 
many treatment as control group members. 

Assume: H 0 : y 2 = 0 

Hp y 2 -A 0 
a = 0.05 
a = 20 

p = 0.57 (correlation between baseline and post test scores) 

Annual study attrition of 1 5 percent from treatment and control group 

We would like our statistical tests of program impacts to have at least 80 percent power for detecting a 
difference in the means of the treatment and control groups of 0.15 to 0.20 standard deviations. 75 That is, 
power = (1-P) = 0.80, and A = |y 2 | = 0.15a in the first case; and A = |y 2 | = 0.20a in the second case. The 
assumptions above regarding test score standard deviations and correlations are drawn from the actual 
data obtained from the previous experimental evaluation of the privately-funded Washington Scholarship 
Fund program, 1998-2001. 76 Though characterized as assumptions, they are likely to be more accurate 
than mere educated guesses because they are based on actual data from a similar analysis. 

Without the aid of baseline test scores, over 1,600 students would need to be subject to random 
assignment for the impact analysis to have 80 percent power in detecting a program effect of 0.15 
standard deviations in outcome year l(Table Al). Accounting for study attrition, the required sample size 
would need to be almost 2,300 students in order to have sufficient power to identify such a moderately- 
size effect in outcome year 3. Should the test score impact of the program be somewhat larger, 0.20 
standard deviations, the analysis would be adequately powered for year 1 if only a little over 900 students 
were randomly assigned and for year 3 if slightly less than 1300 students were randomized. 



75 A number of prominent evaluations of educational interventions have reported statistically significant impacts near or within 
the range of 0.15 to 0.20 standard deviations over one to three years. Such impacts are generally characterized as moderate in 
size and policy relevant. For examples, see Howell et al., The Education Gap, 151; Alan B. Krueger, “Experimental Estimates 
of Education Production Functions,” Quarterly Journal of Economics 114, no. 2, (May 1999): 525; Ann Flanagan, Jennifer 
Kawata, and Stephanie Williamson, Improving Student Achievement: What NAEP Test Scores Tell Us (Santa Monica, CA: 
RAND Corporation, 2000), 59. 

76 See Patrick J. Wolf, Paul E. Peterson, and Martin R. West, Results of a School Voucher Experiment: The Case of Washington, 
D.C. After Two Years, Paper delivered at the National Center for Education Statistics 2001 Data Conference, Mayflower Hotel, 
Washington, D.C., July 25-27. 
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Table Al. Required Sample Size: No Baseline Scores, Balanced Treatment and Control 
Groups 





No Baseline Test Score 
0.15a 


No Baseline Test Scores 
0.20a 


Treatment Group Equal 


Year of Evaluation 


Year of Evaluation 


to Control Group 


One 


Three 


One 


Three 


N t 


821 


1,137 


462 


640 


N c 


821 


1,137 


462 


640 


Total N 


1,642 


2,274 


924 


1,280 



SOURCE: Unnecessary, as numbers are from statistical formulas and not actual data. 



Under the more likely condition of twice as many students in the treatment than control groups, the 
sample sizes required to detect a 0.15 or 0.20 standard deviation program effect after one and three years 
are slightly larger (Table A2). Still, should the test score impact be as large as 0.20 standard deviations, 
the analysis would be able to detect it in year 1 if somewhat more than 1,000 students were randomized, 
and in year 3 if somewhat more than 1 ,400 students were randomly assigned. 

Table A2. Required Sample Size: No Baseline Scores, Treatment Group Larger Than 
Control Group 





No Baseline Test Score 
0.15a 


No Baseline Test Scores 
0.20a 


Treatment Group 


Year of Evaluation 


Year of Evaluation 


Larger Than Control 


One 


Three 


One 


Three 


N t 


1,232 


1,705 


693 


959 


N c 


616 


853 


347 


480 


Total N 


1,848 


2,558 


1,040 


1,439 



SOURCE: Unnecessary, as numbers are from statistical formulas and not actual data. 



Since including baseline test scores in the estimation of experimental test score impacts improves the 
precision and subsequent power of the analysis, and over 75 percent of the members of the initial impact 
sample were tested at baseline, the actual sample size needed for a sufficiently powered analysis may be 
closer to the figures presented below (Table A3). With the aid of baseline test scores, about 2,000 would 
need to be randomly assigned to detect a 0.15 standard deviation effect in year 3, and only 1,100 would 
need to be randomized in order to detect a 0.20 standard deviation effect in year 3. 
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Table A3. Required Sample Size: Baseline Scores, Balanced Treatment and Control 
Groups 





Baseline Test Scores 
0.15a 


Baseline Test Scores 
0.20a 


Treatment Group Equal 


Year of Evaluation 


Year of Evaluation 


to Control Group 


One 


Three 


One 


Three 


N t 


707 


979 


398 


550 


N c 


707 


979 


398 


550 


Total N 


1,414 


1,960 


796 


1,100 



SOURCE: Unnecessary, as numbers are from statistical formulas and not actual data. 



Under the more likely condition of twice as many students in the control than treatment, the sample sizes 
required to detect a 0.15 or 0.20 standard deviation program effect after one and three years are slightly 
larger (Table A4). Still, should the test score impact be as large as 0.20 standard deviations, the analysis 
would be able to detect it in year 1 if just under 900 students were randomized and in year 3 if somewhat 
more than 1 ,200 students were randomly assigned. 

Table A4. Required Sample Size: Baseline Scores, Treatment Group Larger Than 
Control Group 





Baseline Test Score 
0.15a 


Baseline Test Scores 
0.20a 


Treatment Group 


Year of Evaluation 


Year of Evaluation 


Larger Than Control 


One 


Three 


One 


Three 


N t 


1,060 


1,467 


596 


826 


N c 


531 


734 


300 


414 


Total N 


1,591 


2,201 


896 


1,240 



SOURCE: Unnecessary, as numbers are from statistical formulas and not actual data. 
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Appendix B 




Appendix B 

Characteristics of Eligible Applicants versus Nonapplicants: 
DC Public School Students 



The statute requires that the annual evaluation reports on the program include a comparison between 
program participants and nonapplicants from District public schools. The DCPS Office of 
Communications and Public Information provided the technical evaluation team with data that permit 
comparisons such as between public school applicants and nonapplicants as well as participants and 
nonapplicants, using information regarding a modest number of educational and background 
characteristics that DCPS collects on all of its students. Because the only individual signifiers that were 
available in the DCPS database were student names, birthdates, and grades, we were only able to confirm 
the identity in the database of 1077 of 1343 (i.e., 80 percent) public school eligible applicants to the 
program. 77 Thus, the comparisons presented below should be interpreted with caution. 

Chapter 4 of the report includes four tables that compare various samples of program applicants and 
participants to DCPS nonapplicants. Appendix Table B1 provides a fifth comparison, of all eligible 
applicants in the database to all nonapplicant DCPS students, regardless of free or reduced-price lunch 
status and broken out by grade band. Because this applicant sample includes students who did not 
complete the accountability testing and students who were not participating in the free or reduced-price 
lunch program, it is a more expansive applicant sample than some of the more restrictive samples used in 
the comparison tables in Chapter 4. Although this comparison is informative, conclusions regarding the 
comparisons should be made with caution, as the sample of eligible applicants is limited to low-income 
students whereas the sample of DCPS nonapplicants includes at least some middle to high income 
students. 



77 Many of the names of eligible applicants are subject to alternative spellings and the birth date variable was missing some data 
and included some entries that did not appear to be valid. 
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Table Bl. Characteristics of District Public School Students, by Grade Band, Program Applicants Versus Nonapplicants: Spring 2004 





Grades K-5 


Grades 6-8 


Grades 9-12 




mmm 






mmm 












Characteristic 


Osmma 




Difference 


Esmma 




Difference 






Difference 


Baseline Test Scores 1 




















Average Reading Percentile 


46.86 


46.37 


0.49 


39.98 


40.65 


-0.68 


31.66 


32.91 


-1.24 


Percent missing 


43 


44 




3 


6 




4 


14 




Average Mathematics Percentile 


50.95 


51.18 


-0.23 


44.91 


46.11 


-1.20 


43.09 


41.26 


1.83 


Percent missing 


42 


43 




2 


5 




4 


13 




Percent in Special Education 


16 


12 


4* 


18 


16 


2 


13 


15 


-1 


Percent missing 


41 


42 




1 


3 




3 


10 




Percent, by Race 




















African American 


94 


82 




89 


87 


2 


94 


86 


g** 


Other race 2 


6 


18 


_1 1** 


11 


13 


-2 


6 


14 


_g** 


Percent missing 


2 


2 




0 


1 




0 


1 




Percent, by Gender 




















Female 


52 


50 


2 


49 


51 


-1 


48 


51 


-3 


Percent missing 


0 


0 




0 


0 




0 


1 




Percent Participating in Free/Reduced- 
price Lunch Program 


86 


71 


15** 


84 


72 


12** 


81 


60 


21** 


Percent missing 


4 


4 




0 


0 




0 


0 




Sample size 


614 


30,468 




308 


17,279 




155 


19,121 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in tenns of National Percentile Ranks, with 50 as the median score. 

2 “Other race” includes students who were identified as Hispanic, white, Asian, American Indian, or Alaska Native. Because of small cell sizes, Hispanic was combined with other race in this 
table. 

SOURCE: Accountability testing database for DC public and charter schools, DCPS Office of Communications and Public Information. 
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Appendix C 

Characteristics of Student Eligible Applicants by Type of Previous School 



This appendix presents additional detail about the eligible applicants to the program in the initial year of 
implementation. Table Cl describes the forecasted entering grades of applicants, by school type. Table 
C2 presents information about student race and gender. Tables C3 and C4 describe the relationship of the 
adult applicant to the child applicant as well as parental involvement in the child’s education. Finally, 
Tables C5-C8 describe the responses of applicants to various questions about school facilities, problems 
at school, parental satisfaction with school, and the features that parents are looking for in a school. 

These comparisons are merely descriptive in nature. Since students with different characteristics and 
motivations self-select themselves into different types of schools prior to applying for the program, none 
of the comparisons presented here should be interpreted as meaning that the types of schools that the 
students attended actually were responsible for the differences. For example, most applicants from 
private schools seek a scholarship in order to remain in their school, while all applicants from public 
schools are trying to switch to a private school. Therefore, it is not surprising that the parents of private 
school applicants were much more satisfied with their child’s school than were the parents of public 
school applicants. 
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Table Cl. Percent of Students Entering Various Grades, Eligible Applicants: Spring 2004 



Grade Level 1 


Total 


Public 

SINI 


Non-SINI 

Public 


Charter 


Private 


Kindergarten - 1 


20 


10 * 


25** 


13** 


16** 


2-3 


18 


11 


19 


16 


18 


4-5 


18 


9 * 


21 ** 


16 


15* 


6-7 


19 


11 


19 


21 


20 


8-9 


15 


23 


J j ** 


22 ** 


19** 


10-12 


10 


35** 


5** 


12 


13** 


Percent missing 


0 


0 


0 


0 


0 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 
** Significantly different from the average of the other subgroups at the 99 percent confidence level. 

1 Grade levels were combined because cell sizes were too small to report separately by grade. 



SOURCE: Eligible Applicant Database. 



Table C2. Student’s Race and Gender, Eligible Applicants: Spring 2004 



Characteristic 


Total 


Public 

SINI 


Non- 

SINI 

Public 


Charter 


Private 


Percent, by Race 












African American 


95 


96 


96 


98* 


92** 


Other race 1 


5 


4 


4 


2 * 


g** 


Percent missing 


9 


1 


8 


5 


13 


Percent, by Gender 












Female 


51 


51 


50 


49 


54 


Percent missing 


1 


1 


1 


0 


1 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 



1 “Other race” includes respondents who were identified as white, Asian, American Indian, Alaska Native, Pacific 
Islander, or multiracial. Respondents classified as “multiracial” if more than one race category was selected. 

NOTE: Detail may not round to totals because of rounding. 

SOURCE: Eligible Applicant Database. 
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Table C3. Adult Applicant’s Relationship to Student, Eligible Applicants: Spring 2004 









Non- 










Public 


SINI 






Characteristic 


Total 


SINI 


Public 


Charter 


Private 


Percent of Applicants who are the 
Child’s 












Parent 


90 


92 


90 


91 


89 


Other 1 


10 


8 


10 


9 


11 


Percent missing 


3 


5 


3 


2 


2 


Percent of Applicants who are the 
Child’s Mother 


82 


84 


83 


86 


79* 


Percent missing 


18 


16 


17 


14 


21 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 

1 “Other” includes grandparent, step parent, other relative, and other adult. 



NOTE: Detail may not round to totals because of rounding. 

SOURCE: Eligible Applicant Database. 



Table C4. Parental Involvement with Child’s Education, Eligible Applicants: Spring 2004 



Characteristic 


Total 


Public 

SINI 


Non- 

SINI 

Public 


Charter 


Private 


Percent of Parents who Participated in 
the Following Activities with Their Child 
in the Past Month 

Discussed experiences at school 


99 


97 


99 


100 


99 


Helped with mathematics or reading not 
related to homework 


91 


7g** 


93 ** 


92 


88 * 


Worked on homework 


95 


86 ** 


9 ^* * 


94 


94 


Worked on a school project 


87 


76** 


85** 


87 


93 ** 


Attended school activities with child 


89 


65** 


88 


87 


95** 


Average percent missing 


7 


9 


8 


5 


6 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 

SOURCE: Eligible Applicant Database. 
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Table C5. 



School Facilities and Homework, Eligible Applicants: Spring 2004 



Characteristic 


Total 


Public 

SINI 


Non- 

SINI 

Public 


Charter 


Private 


Percent of Parents Reporting that 
Students Have the Following Resources 
at Their School 

Special programs for non-English 


41 


43 


50** 


27 ** 


33** 


speakers 
Nurse’s office 


77 


92** 


92** 


73 


45** 


Prepared Lunches 


67 


§4** 


81** 


64 


40** 


Cafeteria 


80 


97 ** 


92** 


76* 


59** 


Special programs for advanced 


41 


30 


2 y** 


35 


53** 


learners 
Arts program 


70 


53** 


66 ** 


66 


79 ** 


Special education programs 


60 


57 


67** 


58 


47 ** 


Computer lab 


79 


79 


74 ** 


80 


§9** 


Gym 


63 


§9** 


63 


^^* * 


67* 


Child counselors 


79 


84 


82** 


75 


75* 


Library 


82 


85 


85** 


62** 


85 


After-school program 


80 


70* 


80 


75* 


85** 


Music program 


71 


61 


68 ** 


60** 


g4** 


Individual tutors 


44 


27* 


37 ** 


36* 


62** 


Average Hours of Daily Homework 


1.04 


. 68 ** 


.85** 


1.08 


j 41** 


Percent missing 


8 


13 


9 


8 


5 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 



SOURCE: Eligible Applicant Database. 
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Table C6. Problems at Child’s School, Eligible Applicants: Spring 2004 



Characteristic 


Total 


Public 

SINI 


Non- 

SINI 

Public 


Charter 


Private 


Percent of Parents who Believe the 
Following Problems at School are 
Serious 

Fighting 


44 


85** 


58** 


45 


12 ** 


Tardiness 


44 


86 ** 


52** 


44 


23** 


Destruction of property 


32 


79 ** 


42** 


22 ** 


10 ** 


Truancy 


32 


76** 


42** 


31 


10 ** 


Cheating 


23 


^^* * 


28** 


23 


9** 


Guns or other weapons 


18 


54** 


22 ** 


13 




Drug distribution 


14 


38** 


18** 


10 




Drug and alcohol use 


14 


35** 


16** 


11 


g** 


Average percent missing 


9 


11 


11 


5 


7 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 



SOURCE: Eligible Applicant Database. 
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Table C7. Parental Satisfaction with School, Eligible Applicants: Spring 2004 



Characteristic 


Total 


Public 

SINI 1 


Non- 

SINI 

Public 


Charter 


Private 


Percent of Parents ‘Very Satisfied’ with the 
Following Aspects of Their Child’s School 












Amount of information from teachers 


29 




27 ** 


25 


58** 


Freedom to observe religious traditions 


24 




7 ** 


13** 


63** 


Class size 


23 




g** 


15** 


57** 


Safety 


27 




12 ** 


22 * 


60** 


Respect between teachers and students 


28 




13** 


23* 


63** 


Location 


33 




25** 


23** 


58** 


Parental support for school 


28 




16** 


22 ** 


58** 


Academic quality 


26 






21 * 


61** 


Discipline 


27 




12 ** 


24 


61** 


Racial mix of students 


21 




12 ** 


17 


42** 


Average percent missing 


7 




9 


5 


5 


Percent of Parents who Gave the School a 
Grade of 












A or B 


53 


9** 


40** 




88 ** 


C 


29 


26 


37 ** 


37 ** 


10 ** 


D 


11 


32** 


24 ** 


11 




F 


7 


32** 


8 * 


7 


l ** 


Percent missing 


5 


6 


7 


2 


4 


Average Grade Parent Gave School (4.0 


2.50 


1.15** 


2 . 20 ** 


2.36* 


3.33** 


Scale) 












Percent missing 


5 


6 


7 


2 


4 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 

1 Parent satisfaction for students in public SINI schools could not be reported because the cell sizes were too small for 
some response categories 

NOTE: Detail may not round to totals because of rounding. 

SOURCE: Eligible Applicant Database. 
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Table C8. 



Valued Features of Schools, Eligible Applicants: Spring 2004 



Characteristic 


Total 


Public 

SINI 


Non- 

SINI 

Public 


Charter 


Private 


Percent of Parents who Cited the Following 
as the Most Important Reason in Choosing a 
School (Top Four Choices) 1 

Academic quality 


48 


54 


52* 


43 


44 


Services for students with special needs 


5 


5 


6* 


3 


4 


Convenient location 


13 


12 


11 


15 


15 


School safety 


17 


17 


16 


21 


15 


Average percent missing 


30 


25 


32 


27 


29 


Sample size 


1,848 


79 


993 


258 


518 



* Significantly different from the average of the other subgroups at the 95 percent confidence level. 

** Significantly different from the average of the other subgroups at the 99 percent confidence level. 

1 Parents were given a wide variety of reasons to choose from, but only the top four are reported in this table, as cell sizes were 
too small to report for all response categories. The average percent missing is across all response categories. 

SOURCE: Eligible Applicant Database. 
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Appendix D 




Appendix D 

Characteristics of Scholarship Users and Nonusers 



In addition to considering the characteristics of eligible applicants to a scholarship program, it is also 
important to examine the characteristics of the students who actually use a scholarship compared to those 
who do not use one. A total of 1,027 scholarship recipients had used their scholarship to matriculate at a 
private school as of September 10, 2004. A total of 339 recipients had not used their scholarship by that 
date. The tables in this appendix present comparisons between the scholarship users and nonusers on a 
variety of educational, background, and experiential characteristics. Selected items from these 
comparisons are summarized in section 4.4 of the report. The most noteworthy additional findings from 
these data are that: 

• Users were more likely than nonusers to be entering kindergarten or first grade, less likely than 
nonusers to be entering the more slot-constrained grades 7 through 12. 

• Users had spent an average of 6.1 years in their current residence compared to 7.7 years for 
nonusers. 

• The mothers of scholarship users were somewhat less likely to lack a high school diploma and 
somewhat less likely to have a graduate degree than were the mothers of nonusers. 

• Fifty-three percent of users were female compared to just 45 percent of nonusers. 

• Students were less likely to use a scholarship if their parents reported that their previous school 
had special programs for non-English speakers, advanced learners, or students in special 
education. 

• Students were more likely to use a scholarship if their parents described drugs or alcohol use as a 
problem at their previous school. 

• Students were more likely to use a scholarship if their parents listed “school safety” as the most 
important reason for choosing a school, less likely to use a scholarship if they listed “services for 
students with special needs” as the most important reason. 

There were no statistically significant differences between scholarship users and nonusers regarding a 
number of key educational and background factors discussed in section 4.4 as well as mother’s marital 
status and the overall grade that parents gave to their child’s previous school. 
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Table Dl. Student’s Educational Characteristics, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Baseline Test Scores 1 








Average Reading Percentile 








Grades K-5 


49.12 


40.30 


8.82** 


Grades 6-8 


39.82 


38.06 


1.76 


Grades 9-12 


30.00 


23.57 


6.43 


Percent missing 


56 


41 




Average Mathematics Percentile 








Grades K-5 


53.47 


43.75 


9.72** 


Grades 6-8 


44.91 


42.26 


2.65 


Grades 9-12 


40.30 


43.83 


-3.53 


Percent missing 


56 


40 




Percent with a Learning or Physical Disability 


9 


29 


-20** 


Percent missing 


6 


8 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Test-score results are in terns of National Percentile Ranks, with 50 as the median score. 



SOURCE: Eligible Applicant Database. Test-scores obtained from the accountability testing database for District of 
Columbia public and charter schools, DCPS Office of Communications and Public Information. 
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Table D2. Percent of Students by Forecasted Grade Level, Users and Nonusers: 
Spring 2004 



Grade Level 


Users 


Nonusers 


Difference 


Kindergarten 


14 


9 


5* 


1 


13 


6 


g** 


2 


11 


9 


3 


3 


12 


9 


3 


4 


11 


9 


2 


5 


11 


13 


-2 


6 


8 


9 


-1 


7 


7 


13 




8 


7 


11 


.4* 


9 


2 


4 


-1 


10 


2 


6 


_4** 


11-12 1 


1 


4 




Percent missing 


0 


0 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Cell size too small to report separately by grade. 



SOURCE: Eligible Applicant Database. 
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Table D3. Family Income, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Students whose Family Income is 








Less than $5,000 


11 


11 


0 


$5,000-$10,999 


18 


24 


-6* 


$1 1,000-$24,999 


43 


38 


5 


$25,000-$39,999 


25 


24 


1 


$40,000 or more 


3 


4 


0 


Average Family Income 


$18,652.49 


$17,840.70 


$811.79 


Percent missing 


0 


0 




Percent of Students whose Families Received the 








Following Forms of Government Assistance 








Total wages, salaries, tips 


53 


44 


9** 


Housing Assistance 


7 


6 


i 


TANF (Welfare) 


22 


28 


_7* 


Supplemental Security Income 


3 


6 




Social Security 


10 


14 


-3 


Child Support 


8 


8 


0 


Gifts from family/ffiends 


1 


1 


0 


Interest and dividend income 


1 


2 


0 


Other income 


7 


8 


-2 


Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



SOURCE: Eligible Applicant Database. 
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Table D4. Mother’s Employment and Residence, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Students whose Mothers are Employed 








Full time 


46 


44 


2 


Part time 


15 


15 


0 


Looking for work 


30 


30 


-1 


Not looking 


10 


11 


-1 


Percent missing 


13 


23 




Mother’s Average Years Work Experience 


7.81 


7.98 


-17 


Percent missing 


51 


57 




Average Years at Current Residence 


6.11 


7.69 


-1.58** 


Percent missing 


2 


3 




Percent who Rent and do not Own 


86 


82 


3 


Percent missing 


14 


17 




Monthly Rent/Mortgage Payment 


$618.38 


$586.36 


$32.02 


Percent missing 


5 


5 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

NOTES: Detail may not sum to total because of rounding. Mother response is for biological parent only. 



SOURCE: Eligible Applicant Database. 
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Table D5. 



Mother’s Education, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Students whose Mothers Attained the 








Following Level of Education 








No high school diploma 


11 


19 




GED 


6 


6 


0 


High school diploma 


25 


25 


0 


Some college 


35 


30 


5 


Vocational (2 yr) degree or certificate 


15 


14 


1 


College graduate 


7 


4 


3 


Graduate degree 


1 


2 


-2* 


Percent missing 


11 


19 




Average Years of Mother’s Education 


12.73 


12.50 


23* 


Percent missing 


11 


19 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



NOTE: Mother response is for biological parent only. 

SOURCE: Eligible Applicant Database. 
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Table D6. Mother’s Marital Status and Household Composition, Users and Nonusers: 
Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Students whose Mothers are 








Single, never married 


61 


58 


2 


Married 


19 


16 


2 


Divorced 


10 


13 


-3 


Other 1 


10 


12 


-1 


Percent missing 


10 


18 




Household Composition 








Adults in household 


1.33 


1.31 


2 


Percent missing 


0 


3 




Children in household 


2.83 


2.70 


12 


Percent missing 


0 


3 




Household size 


4.10 


3.94 


15 


Percent missing 


0 


0 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



“Other” includes widowed, separated, and divorced, remarried. 

NOTE: Mother responses is for biological parent only. 

SOURCE: Eligible Applicant Database. 

Table D7. Mother’s Ethnicity, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Students whose Mothers are 








African American 


93 


93 


0 


Other race 1 


7 


7 


0 


Percent missing 


15 


22 




Percent of Students whose Mothers are 








Hispanic (any race) 


8 


7 


1 


Percent missing 


10 


18 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



1 “Other race” includes respondents who were identified as white, Asian, American Indian, Alaska Native, Pacific 
Islander, or multiracial. Respondent classified as “multiracial” if more than one race category selected. 

NOTES: Detail may not sum to total because of rounding. Mother response is for biological parent only. 

SOURCE: Eligible Applicant Database. 
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Table D8. Student’s Ethnicity and Gender, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent, by Race 








African American 


94 


96 


-2 


Other race 1 


6 


4 


2 


Percent missing 


6 


6 




Percent Hispanic (any race) 


7 


7 


1 


Percent missing 


4 


7 




Percent, by Language Most Spoken in Home 








English 


93 


96 


-3 


Other language 2 


7 


4 


3 


Percent missing 


4 


4 




Percent, by Gender 








Female 


53 


45 


9 ** 


Percent missing 


1 


2 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



1 “Other race” includes respondents who were identified as white, Asian, American Indian, Alaska Native, Pacific 
Islander, or multiracial. Respondent classified as “multiracial” if more than one race category selected. 

2 “Other language” includes all other languages including Spanish and Amharic. 

NOTE: Detail may not sum to total because of rounding. 

SOURCE: Eligible Applicant Database. 
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Table D9. Parental Involvement with Child’s Education, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Parents who Participated in the 
Following Activities with Their Child in the 
Past Month 








Discussed experiences at school 


99 


98 


0 


Helped with mathematics or reading not 
related to homework 


94 


90 


4 ** 


Worked on homework 


97 


92 


5** 


Worked on a school project 


86 


85 


1 


Attended school activities with child 


90 


85 


5* 


Average percent missing 


8 


6 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 
** Statistically significant at the 99 percent confidence level. 

SOURCE: Eligible Applicant Database. 



Table DIO. School Facilities and Homework, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Parents Reporting that Students have 
the Following Resources at Their School 








Special programs for non-English speakers 


37 


54 


_ J7** 


Nurse’s office 


80 


89 


_9** 


Prepared Lunches 


70 


77 


_7* 


Cafeteria 


81 


90 


_9** 


Special programs for advanced learners 


33 


46 


- 12 ** 


Arts program 


67 


72 


-4 


Special education programs 


57 


73 


-15** 


Computer lab 


75 


79 


-5 


Gym 


58 


68 


- 10 ** 


Child counselors 


78 


84 


_7* 


Library 


80 


86 


- 6 * 


After-school program 


82 


78 


4 


Music program 


69 


72 


-4 


Individual tutors 


37 


45 


- 8 * 


Average Hours of Daily Homework 


.93 


.96 


-.04 


Percent missing 


9 


7 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



SOURCE: Eligible Applicant Database. 
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Table Dll. Problems at Child’s School, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Parents who Believe the 
Following Problems at School are 
Serious 

Fighting 


50 


46 


4 


Tardiness 


48 


48 


1 


Destruction of property 


35 


35 


1 


Truancy 


34 


39 


-4 


Cheating 


26 


22 


4 


Guns or other weapons 


20 


15 


4 


Drug distribution 


16 


10 


6* 


Drug and alcohol use 


14 


10 


5* 


Average percent missing 


10 


9 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 
** Statistically significant at the 99 percent confidence level. 

SOURCE: Eligible Applicant Database. 
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Table D12. Parental Satisfaction with School, Users and Nonusers, Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Parents ‘Very Satisfied’ with the 
Following Aspects of Their Child’s School 








Amount of information from teachers 


26 


23 


3 


Freedom to observe religious traditions 


19 


13 


6* 


Class size 


18 


17 


1 


Safety 


22 


20 


2 


Respect between teachers and students 


23 


21 


2 


Location 


31 


25 


6* 


Parental support for school 


23 


24 


0 


Academic quality 


21 


18 


3 


Discipline 


22 


20 


2 


Racial mix of students 


17 


15 


2 


Average percent missing 


8 


9 




Percent of Parents who Gave the School a 
Grade of 








A 


19 


15 


4 


B 


29 


31 


-3 


C 


32 


32 


0 


D 


13 


13 


0 


F 


8 


9 


-1 


Average Grade Parent Gave School (4.0 Scale) 


2.37 


2.30 


7 


Percent missing 


6 


6 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 



SOURCE: Eligible Applicant Database. 
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Table D13. Valued Features of Schools, Users and Nonusers: Spring 2004 



Characteristic 


Users 


Nonusers 


Difference 


Percent of Parents who Cited the Following 








as the Most Important Reason in Choosing a 








School (Top Four Choices) 1 








Academic quality 


49 


47 


2 


Services for students with special needs 


3 


13 


-10** 


Convenient location 


13 


11 


2 


School safety 


19 


11 


g** 


Average percent missing 


30 


31 




Sample size 


1,027 


339 





* Statistically significant at the 95 percent confidence level. 

** Statistically significant at the 99 percent confidence level. 

1 Parents were given a wide variety of reasons to choose from, but only the top four are reported in this table, as cell sizes were 
too small to report for all response categories. The average percent missing is across all response categories. 

SOURCE: Eligible Applicant Database. 
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